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^ j^l^ough the persons altehding thMrivitationalj ^WMfcce ori Test- 
ing; problemi have a commoh boiidf^hey repretontt^miy dinerent 
intfi^tg iri the jBeid of meaiurement. These range jfart the guidance 
cpunielor to the test dey eloper, fVorn the pure resep: ' ^ 
administrator, from the psychologipi to tho.stEaisli 

: Tljjis diverlity bf interest presents a Vudl dmlleng^ 
Chairnian— how to atruclure 0 prog^r^ 
. Jnterert of this group piid yk enable eaed parlic i 

f^A^ io mrry m and pfaetiees that nrr 

, filimulating find usefulP " ' 

/ Tlie success of the 1057 Invitationa! Conierericirt^csts to the niartriur 
in \^ich Chairman AHliur ^T^ Beset vj^ith all 

the adrnimstrative problems of his own, annuar Ec^cational p4c6iy 
BOriittu Conferences he, iieWrl^el^gs,'^ siicceeded in 'scheduling^ for theV 
Inyitatlorial Conjerence, on/tHrfbllowing'day, a stimulating series oT 
reports and discussions on thi theme *4rnprovir^ the Quality and Scope 
bf Measurement." Tlib lai:ge numbuivof persons In the anaience felt 
well-rewarded. . . " , 

These few words, cann^ adequately e^tpress my ^ appreciation foi: 
Arthur TraxlerV elltorts, for the exceU of each of the 

speakers^ and for the efficient mahher ln which the Conference was 
conducted. I hope these PrQceBdihgs vvill serve, to give ati even wider 
aumence the full flavor of an important conferqnce 



IIr^hV Chaunc«y 
% P res i din I 




b 



The .1957 Invitatlorial ponference on TeSiing ProBlenis, wjiich was! 
held at l^e IJptel Roosevelt oh ^ IpSTy was Qdiiderned With 

the hroad tHeme *7^pro\^ng/the Quality iand Sco^e of MeaiUt6ment/V 

mhjor tnea|uremerit " heeds, ,1^ bot|i the eaucational and Ivocati 
araasj denionstmled progresi is likely to depend upon 

criteria of iuceesp more adequate than;niftrks or^ove^-aH ratings. Where 
criteria art obscure in n^eanlhg or; limited in scope, iihprov.ertient jn 
tests may not be reflected ill higherp dorr ek^ with the criteria. In 
ft^ct^ it is^jf^tirely ppiiible fokb6tt0^ ^^^^ to^be related to a lesser dejr^ 
with faulty 'qriteriai,^ fLj^^-:. ' 

In ,an eflbrtw to stimulate: ihnilting- and discu^i^n^ tliis problerp 
. §niohg test speqi^lists, the flrst sesdion of, the Invitational Confcrenib 
was cbncerried with the topic "Improving Criteria for Educational and. 
Psychological Meafiurement,"' with^paperE on **Criteria for Complex 
/Mental Prwesses^' by Robert G. Wikon of Reed College and the Gifted 
Child Project* Portland (Oregon) Puflhc Schools, and on "Criteria of 
Nonintellectual Aspecji of Personality" by Morris L Stein of the Uni- 
verdty of Chicago. , ; , 

Another major need for progress in educational measurement is ini- 
provemerit in techniques of writing test exercises. This improvement 
may come about in two ways. In the first place^ test constructorii may^ 
devise new and novel situations on which test items are based. There 
might be more experimentation with Jistemng comprehehsioni with 
laboratory deTnonstration followed by items calling for integration of 
ideas gleaned from observation^ and so^forth. In tjiia second place, the 
quaUty of the conventional kind of item may be raisbc/ through improve- 
ment in the content and clarity of each item wfitten and through in= 
genuity in devising some wholly difFerent lunds of tpst items. 

The second conference session considered the queatipn of "Improving 
Measurement Through Bjptter Exercise Writings with stimulating 
papers on "Exercise Writing in the Field of the. Humanities*' by Paul B. 
Diederich of ^Educational Testing Service, "Exercise Writif^g in the 
Natural Sciences" by Leo Nedelsky of the University of ^.IHqago, and ^ 
"Exercise Writing in the Social Sciences" by Max D. Engelhart of the 
^Chicago City Junior College. ^ 4 i f 

A third measurement-need, one. which has-been referred to with 
increasing frequency in recent years, is better Gommunication between 
test specialists and test consumers. Test specialftts need to convey the 
basic concepts of test construction, standardi/.ation, validation, and^ 
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' application in language that test users tun uncjerstand. Bu^ equfllly, 
. l^rtant, if Isdeairable to rcgicmber that idSas flow iii bdlli^|rectlorift: 
and that th^ measureraent fraternity can Icnrn mupli from Uio prac- 
' titiqnera— the schooJ ddmpstratora/ the" teucliers, and th«> gnidanco 
counselors, » • . I . , ' . ..n. 



'TheTfftWiroon^eiBioh-tjf-the-t^ 

Users' Problenis as Guides to Bettor Aft'iisurenient.'' "1 ho Sdiofil 
Adminiatrator/ Problems for VT4stera'1.vvLM by l'«ul T. 

Raiikin of th/ Detroit PubliwSclioois. and |us piwor %vaa sjisf ussed by • 
Roger T. 'Leniioh of the WoM liook C:»riipirti^ "Tim UuidiUir^; Diroo- 
. idr's Problems and Suggestfcris |oT7tere|t SpftualisJ" ^re preseiited 
by Edl^ard Landy of the TOsviltni (MasWfiusL-Us) I'ublw. S^iools, %vilh 
• discussion by Donald E. Super of TeaclipnSQ^^ ColutnlJiH riiiveimt y. 
: " An outstandi^ fenlliru of llit'^ujiift^cnci' wiis ihg l^theoii address' 
hy Emeritus Pwifcssor l^dwa* K. Strtiiig. Jr.. of Slaftford I luvursily, 
■ ton "Prediction hi EdutuiUouiflli W Vot'Lilioiiul Kuccl'ss Throiigli Inlerest 
_^j£asurem|nt,'^a neld to wliiofl Dr: Sirnrig hi|. d(wolod iMifetimo of 
profossibiul leatleryrriJ^ a ^ 

The conference was utiepded by over Sf)iy'cU;legates, jnntndmg a large 



proportioirfof the iuc;a£Mfenl^eci»^ls .rilM^'iiiU'd States u.W n 
number froni other counlrit's as wjhf*^ ' v ', ' 

. , ^ • T^*^ Aktuiim K, TuixyiH 

Chairgmn 
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|E^SION I 



Improying* Criteria for Ediicational 
. and Psychological Meiisurement ^ 



^0 



Remarks of the! Ghairmaii ^ 



JOSHUA A. FISHMAN 



Together with many of you I attended last year's Invitational^jCon- 
ference on Testing Problems. As always*- this was a most rewprding 
experience an^ one which provided m^i food for further' thought* 
discussion, and study after the Conference was over. Tlie over-all theme 
of the Conference, as rnany of you^will remeniberi was "Testing— Then 
and Now" and the program prRsented a magnificent jp^nofaJ^ia of the 
progrfis (and fllsot quite frankly, of the lack of progress) of the ,testin^ \ j 
rafovement during the past several dec^fpii > ^ 

One pf^e major point s implied by last year's Conference was the 
itill far from satisfactory state of affairs— both conceptually and opera= 
tionally— in conjunction with Uie entire area of test validity, ^ur major 
satisfactions in testing work must conie from other sources. 

During the past quarter century we have vastly improved our ad- ^ 
ministering, pcoring and reporting techniques. As a result, we can how 
calmly undertake testing programs of a magnitude that would have 
sttfggercd the imagination only a few generations ago. Into the College 
Board testing program (with which ^ am most intimately familiar) a 
new test was introduced a short two years ago. In its very first admlnis' 
tration it reached 100,000 individuals, and this year it has reached well 
over 300,000 individuals, (Both of these figures refer to a sing^riay, ' 
nationwide administration calling upon the integftited cooperation of 
a veritable army of ipecialists of y^ious kinds.) Certainly our ability 
to perform miracles of this kind has improved markedly in a relatively 
short time span. Qur host at today's Conference is a prime example* of ' 
trulx^un matched and quite unbelievable acconiphVhment in this area — 
as in every area relating to testing. > ' 

A iecond major front of our progress has been in the crucial area qf^ 
test reliability. Here we have pelfect^d^-rarious conceptual models as 
well as operational procedures for^^ppropriat|lj estimating and suc- 
cessively improving the dependability of -our measures. Our extensive 
work in scaling and in item analysis has slowly whittled away at test 
ir reliability, especially in the classical achievement and aptitude areas, to 
the extent that test reliability is currently one of our chief sources of 
professional pride. \ 
^ This brinp us to the c^ntralissue of validity^ Here our progro^ has 
been least satisfactory. The nnmber of toHl8 grows from year ib year, 
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theTiumber of individuals tested rises continually, the reliability of the 
. ^ores^ ob&|ned i^appr^iabiei ^tfi^ yet we are often embarrassed to 
^ admit that the criteria against which we validate these scores are tre- 
quently gro^ or ambiguous— at bast partialr&ild^~wbrst, npii-extitTOtf 
The corr^tion of this state of affpim^ a diflRcult tq^k and one.tbVych 
all of us— ETS, and everyone of us ^a.biwlds or uses tests— must con- 
gt^ntlV ireaddresa diirsely perhaM^b^inajor testing' frOTtier 

for the remaindi^ of this cientury: the sel^tion and corablnation of 
criteria apprdpriate to the subtle nuances represented by our test titles 
.on the ode^hknd and*appropriate also to the comple^ties of the si+iLiii- 
tions for which teM behavior is suisposedly prMictivorpn the other 
'We are fortunate indeed to have an opportunity to listen to 
speakers who ifave done considerable work in research area for which, 
the criterion problem is particularly difficult. Our firit speaker. Dr. 
Robert Wilson, appears before us in the dual capacity of Associate 
Professor in the Department of Psychology at Reed College and Re- 
search Director df the lifted Cliild Project of the Portland (Oregon) 
Public Schools. Dr. Wilsons topid is '^Improving Criteria for Con> 
plex Mental Processes.", Our second spehfcer, Dr. Morris I. Stein, is 
Associate Professor of PsycEoIdgy at the University of Chicagq. Dr. 
— Stein will talk on "Criteria of Nonintellectual Aspects of PersonaUty/' 
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• Improving Criteria ' for 
Complex Merjtal Processes 



R.OBERT C. WILSON 



^ The. inclusion of the lopic "Improving Criteria for^ Complex Mental 
Processes" in today's pro^arn is in part ari ea^re^ion'of the feelings 
of frustration experienqed by test-deyelopers. At^east two factpss jiave 
€ontrib%tetf to these feelings ":of frurtration: fifst, the feelings on the 
part of test-conatructoii that they are actually Improving tHe^fediotive 
power of tlleir measures of aptitude and achievement, aceompanied by 
an awareness that jthey have very little actual evHqnce that the improve * 
ment is substantial and a feeling that much of the difliculty is due to 
^the crudeness and lack of senBitivity ifi such time-worn criteria as school 
or college grades; secondly, the criticisms expressed by test-users that 
the tests whiph are available measure or predict only simpk mental 
processes and fail.lp supply sound measures of the inore important kinds 
of mental fui^tionihg sucli as are involved in critical analysis, synthetic 
thinking and the various credtive proceeses. These latter criticisms are 
^in turn, perhaps, 'an expressio» of the feelings of frustration of test-users 
who are faced with the probyhis of measuring and predicting in'complex 
situations. f 'V ^ 

May I add my own feeUngs of f nistration as a test-constructor and 
test-user concerned with t^e' problem of identifying "complex mental 
processes'* in action and with tlR* problem of observing thern in a syste- 
matic manner. We have receijtly completed in' P^tland a five-year 
experimental study for tfifc purpose of developing better educational 
experiences for both intellectually gifted and talented children. The 
result of these five years of experimeritation have led to the develop- 
mtfnt bf a variety of special educational provisions for gifted children. 
Tiiese provisions were adopted last spring by the Portland School 
.Board as a permanent part of the educational program of the Portland 
Public Schools and are being expanded to include all of the schools in 
the Portland Public School Sys^^^l (1). 

The initial proposal to the'Fund for the Advancement of Education 
in April 1952 requesting, support for the study outlined five essential 
features: / ^ ' 

L Provision for many kinds of unusual ability so that the traits and 
. talents selected for identification and for development shall not 
be Umited to generfll intelligence as currently tested and shall 
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include creative* intellectual, artistic, and social cupacities, and 
the emotional and moral qualities necessary for e{Toctivo use 
these capacities. i 

2, Experimentation with metliodg and materiiiis oi' instruction for 
groups and individuals that will challenge and deyelop unusual 
abilities of various kinds, and^to tln!s end the enconrageineQt and 
training of good teacher^. ^ 

3, Coordination of the teachiiig and the prognuns of promising 
students with the coinmon curriculum of the. schools and witli 
other educational resources In tlife communtty to avoid fixed group-'^ 

' hig; with the intention of enabMng other^ students, (and in sonic 
' / ineasurd all %tudent^), to profit from the experinientation. 

t. Cooperation witli ortfer- colleges for folloWiiig up the studeiiti from 
the program aiM foH working out closer articulation of college 
curricula with those of the higli schools, and possible ac- 
celeration at either the high school or college level or toth, 
5. Close colfeboration with a college of liberal arts and sciences in 
a .strategic position for assisting in slnipiTig aii^l evaluating the 
program and for actively participating in important a^ipectB of it. 
' The last^ three features of the program have been relatively ensy of 
accomplishment and evaluation. The fftt two have been more difficult 
^nd relate olpsely to^our topic today. Let us analyze them <^e at iHrnu' 
and expand their meaning. Proposed feature number om Was: "Pro- 
vision for many kinds of unusual ability so that the traits and talent, 
selected for identification and for development shall not be limited to 
general intelligence as currently tested and shall include creative, in- 
tell^tual, artistic, and social capacities, and the emotional and moral 
qualities necessary for efTective use of these capacities." Let us omit 
the last two characteristics, *'the emotional and moral qualities neces- 
sary for efTective use of these capacities," since they lalt in the realm 
Vf Dr. Stein's paper on. "Criteria of Nonintellective Aspects of Per- 
sonality," and turn our attention to the first mentioned charafaeriBtics. 
that is, '^creative, intellectual, artistic, and social capacities." 

The identification of those pupils possessing the general intellectual 
qualities which are related to school success presented no great problem. 
We have used a number of standardii^ed testK of aptitude and achieve^ 
rnent in combination with teacher obHervalionH of classroom perform- 
ance We did encounter diflriculties when we turned our attention to such 
complex processes as ''creative, artistic, and social capacities." 

It was decided at the beginning of the study that we would attempt 
to identify the ten per cent most inlrnted students in each of seven 
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talentB. Thim were art, music, creative writing, cr«aiive draiiiirtloH, 
creative dance, social leadership, and niechanical talent. 

To identify the ipecial talents we astabUshed conimitteqs of leacherH, 
^dmiiiistrators, supervisors^ hnd lay people from the community who 
had technical training, ability, and interest in these talent areas. These 
committees examined the existing testing devices in the various fields, 
and in most cases found that there were no adequate testing devices 
available. The committees then proceeded to develop tlMr own exercises 
for screening purposes in these various talents. In most instances they 
turned in the direction of developing performance tests. We now have 
available seven sets of scrceiiirig exercises, one for each of the taleut 
areas mentioned earlier. These exercises are not all given in any single 
grade, rather tliey are distribntey over several grades to avoid overload- 
ing ally single gpade. Ilowever. the exercises fqr ^ach talent are ad- 
niinistered in at least two grades. By way of example, I will describe 
one of the screening devices, that for creative writing, in some detaih 

The creative writing screeriing device consists of iiyv exercises whicli 
are admiixistered to all fiftli and sixth grade cliildren. The first exercise 
is concerned with Developing Expressive Sentences." The teacher 
gives the children some preparation for this exercise and then on the 
day of the test gives the children sentences like this, for example; "Tin* 
man went down the street."' The children are asked, "In what way (•onld 
you add to or change the word man to give a clearer picture of the ma m^ 
In what ways could yon change other words in tlie sentence to make us 
see this man going down the street?" Here is an example of what one 
child did witUthis. he chaiiged *'the man went down thf street" to ''the 
blind man with the white cane tapped his way down the busy street.". 
The children are given several such sentences, to develop and expand in 
order to make -them more expressive. The results are rated by llic^ 
teacher on a five point scale in terms of certain criteria. 

The second exercise is ''Developing a Paragraplt^tkan a Sentence,". 
The cluldren are given some preparation for this rftid then are given 
severaftentences and told toidioose one and write a paragraph about it. 
For example, *"The mysterious box drew nil eyes to it/* or ^*A_fter all 
it was only a mouse," 

The third exercise is "Writing* ^ .Story Irom Descriptive Phrases. " 
The teacher writes on the board Several descriptive plnases, such as, 
'4iigh-fenced yard,'' ''yelling crowd," ''abandoned mine," ''stormy sea." 
'faithful horse," etc. The children are to select three of these phrases 
and develop a story using the descriptive phrases in the story. 

In the fourth exercise, "Writing an l^xperience." the children are 
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a^ed to tell the most exciting or amusing experience, imaginaiiyc or 
real, they have had in the past few months. ^ 

The fifth exercise is writing an imaginary conversation between two 
characters from fiction the child has read. 

Exercises similar to these are used in several of tlie other taleni areas. 
They are generally fatedby the teaclier, which has some h!rnitations but 
also some advantages. I would hke to quote from a statement of the 
philosophy behind these screening exercises: *'Since the nun\ber of 
talented children wUl vary from class to class, and since the standards 
of rating will vary from teacher to teaclier, it is impossible to specify 
that a particular number of children be regarded as talented or that a - 
particular score be Tegarded as the miiiimum eulofl\ As a general guide,^ 
however, it is^ likely that on the average there will be one to three out= 
standing pupils in a class. In some classes, however, there will be none 
and in some exceptional classes there may be W many as six or eight. 
In looking at the total scores, the teacher may notice a few high scores 
close togethe? with a^ larger gap between, these and the next highest/ 
scores. When this is the fease, the gap may be used as the cutolT point 
and the children above this point may be tentatively identified. 

■'In interpreting the results of the talent screening devices it is ini- 
portant to bear in mind their purpose and the uses to which tliey may 
be put. Their purpose is to help us pick out tliose children who may 
benefit most from further work in the taltnt being tested, Tliey may be 
used to enable us to achieve more closely .the ideal which is often ex- 
pressed of ^meeting the needs of each child.' 

"The purpose of the identification exercises is not the assigning of 
labels. They should by no means be regarded as a certification device 
or a stamp of approval. Tlie screening devices and the administrative 
procedures used are only a %ugli screening procedure. They are fiir 
from perfect. We may ideiitify as prouiising some children who will 
not live up to this promise in further work. We may also miss a few 
children who actually do have promise, 

"Our hope is that by providing a standard set of exercises, all children 
will have an opportunity to show their abilities and that those selected 
as outstanding may have an oppoftunity lur further development of 
their abilities. By repeating the exercise at several graders, we may 
have indreasuig confidence in our selectioii if a child continues to show 
up well.'" 

The preceding is an exaniple of one attempt to develop tests for 
certain complex mental prc^esses— tlie development of informal tests 
disguised as ordinary classroom exercises which may be used to elicit 
teacher judgments about who should receive special attention in each 
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«)r«everararens. This Imn provod to bn a useful npproaclh Pupils Helertccl 
fur special clas^eB on tliu basis of tlim' scronuliig nxeirisos do well in 
lliu classes. 

Oin' second ninjor obji|('tivc ^nc^ludod the staleinent, Kxperinienta- 
lion with methuds and nialerials of in^trncUon for groups and indi= 
viduals that will clinlluugu and develop uiuisual abilities of "various 
kinds. " A variety oF special provisions liave.bt'en tried out, aiid I dc) 
not have time tiere to dem^ribe therii ylL . . ^ ^ 

Of/e of the methods taken ((iT elaborate this olijertive was the eHtablislT- 
inent of seminars in the high schools. -Sennnars are olleied ni literature, 
social studieSj mathematics and science, '^liie seiniiiar is here delined as a 
.small group of students, witi/ one or niore teachers, following an un- 
prescribed, elective course, the coiilcnl of w hrch g(H*s above and be^Hiiul 
the oirerings of normal classes and ^ernphuHi^eH Hull-direeted stuxly, the 
c ri lica I ex a m i na t i o n a i id in te i-p re La t io 1 1 o f oorn u k j n rea d i n gs, a iid pen e= 
trating class discussion. Among the aims of such seininars are those of^ 
(1) making students niore intellijient and pcreeptive readers, (2) inn 
prdving student writirig. (3) developing the'abilily to do iudepehdent 
research work, (4) developiug tin' ability to think critically, and (5) 
developing the ability to think creatively. These all involve eomplox 
behavior and coniplex mental processes. While we have used a v;iriety 
of formal and infornial rt hhu is of ova Ina ting tiie aehievemenl of these 
aims, we are not satisfied with the teeluiiques we have used. 

Turning now from, our own particular problems, in Portland, I should 
like to indicate two major directioiis in wliirh I believe progress will be 
made in improving our undejgtanding of and our ability to measure 
complex mental processes and sonH^ examples of work now being done 
in these directions. 

One pf'ttiese directions is that of clarifying oar c^qnveptualizalkms of 
Ihe complex menial processes so tliat tlie tests w^e develop of tliese proc- 
esses will Iiave greater o^)ntent validity and will elucidate our under- 
standing of luunan mental behavior. The other direction is that of 
specifying more carnally siiualional faciors which relate to the pre- 
dictive validity of any particular applieatiqii of a testing device— an 
elaboration or analysis of the niulti=dctgrmined criteria which are what 
we usually have available. 

In regard to the first direction, one may observe that nuich progresH 
in science has resulted from the clariflcatio!i of concepts through ttie 
substitution of more precise terms for certain commonsensicak popular 
or global terms. Two efforts wliich seem to me to oO^er useful possi- 
bilities in the development of new concepts and new distinctions are 
those reported by Guilford and Bloom. 

^ ^ 17 
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Guilford has been directing since 1949 a series of studies of ap^tudes 
of high-level personnel under contract with the Office of Na^al Research. 
These studies have had as their starting points four global and common- 
sense concepts— reasoning, creativity, planning, and evaluation. The • 
general plan of attack has been, first, to develop hypotheses concerning 
more specific abilities which might gO into these global ijoncepts. Then 
a variety of tests are developed to' measure aUe.rnative conceptions of 
the various specific abilities. The tests are then administered to large 
numbers of individuals and the results-factor analyzed, leading to the 
acceptance or rejection of alternative liypolheses. During the eight 
years in which +hcBe studies have been going on a number of factors 
discovered fcrbthof invustigators have been confirmed and a number 
of new ones have bcnn diacovered. In all, ; about forty factors seem 

well established; > , ' • ' r 

Guilford has noted certain, regularilit^ii 'the tests of these factors 
and on the basis qf them lias devcloped#tiieoretical model of intellect. 
Tins 16 described in detail In two^of his recent publications (2, 3). In 
brief he first divides iiitellec^ito memory and Ihinkmg. lie further 
divides the thiriking abilities hito three categories 'under the helpings 
of cognUion, ppodudion, and evalualion. The co|nitive abilities aje , 
defined as those which Imvc to do with the discovery of information; 
the productive .abiUtics hnvc to do with the use of information; and 
the evaluative- abilities have to do with dccisidns as to the goodness, 
accuracy, or suitability of infonnalion or products. He further sub- 
divides the abilities listed under production- into those involved in 
thinking thai converges on one rigiit answer and those Jnvolved m 
"thinking that diverges or goes off in different directions. The latter is 
most cl6sely related to creativo thiiiiirng. f _ ' 

The tests under all of these classifications arc furthtar categorizecl m 
terms of their figural, structural, or conceptual properties. Guilford^ 
studies have already provided a wide variety of new tests many'of whi6b 
are presently being used by other investigators, llie theoretical model 
is suggestive of still other tests and abilities and further studies- which 
may clarify our understanding of cotiiplex mental procesBes. 

A second effort, which 1 would like to mentioii, in the development 
of new concepts nnd now distinctioils among compjex menial processes 
Is the work of a committee of college and Murvcrsity examiners _ 1 lie 
results of tlieir deliberations have appeared in a book entitled A 
Taxonomy of Educalioti Objectives, Handbook I: Cognitive Domain 
edited by Benjamin S. Blootn (4). One of the purposes of the book is 
to provide a hierarchical classification of measurable educational ob- 
jecUves in the cognitive area. Oognitive is uscd^to include activities 



la 



18 



ROBERT, WitSON 

such as remembemngand i ecalliiig; knowledge, tiiitikiiig, problem solving, 
and creating. ^ , 

Thebodi includes descriptions in lenns of cluuiges iii Htudenft behavior 
of six major categories of educational olyectiveB in tlie cogiulivtj doniain. 
The lowest category is kn^Mge klefined as the recall of ^ecifics and 
univ^rsals, the recall of methods and processes, or the recall df a paMcrn, 
structure, or setting. Parenthetically, ^ is recognized that tlws type of 
educational oyectlveMs *the easiest to faeasunv and ihercfoi:e' the one 
most often emphasized in current riurrtcula/ The category at the next 
lowest level is called compretefi^^on wkich.is defined as^|ll?udiiig those 
objectives, bohaviors,, or= responses whicli i-cprbsen t ari nnderslandiiig 
of the Hteral nibssage contained in a copi:iiHini('ati()y, The iiexnevel is 
mllednpplication aii^l iiivolves the use af^ nbslrncMHtiij^ tn particular and 
concrete situations. At tlie fourth level is/a'cntn'gbry 'caYled iinai}isis 
which involves thefcreakdown of a cpmmunicaitioirrtfft) its cunNtilfioiil 
elements or parts such 'that the rclativq^er^p^iy of "ideas is fnade clear 
atid/or the relations between the iduas^iflp^ed.jj[R?"nm explicit. The 
fifth level is called synihesis which Is ide^^JMajJ ijiC'^ntiing rogether of 
elements aiid parts so as to fovimn whtjjij^^aiiir sixth level is rdllcd 
evalmtion which involves judgrnejitH. abbiiC Iji^ N^ahu^ of riinterial and 
methods for given purposes. Each Qf tluf^fil&jor nibgorie is divided^ 
into sub-categories on the basis of obaervalvUr diiler^nces in stiident 
behavion " j;^:.^^ )*#^ , 

^ The book includes descriptions <)]' ,s^nlpllrol)i(•(•!^'es arid ae^tivitieS' 
under each category, nist) namph^ \ml \\rniA i\)V atttMuplirig to niensunr 
^ each category nntl a (hHiMw^ori ol' Iho chnnu^lcrisllj^H of test ilcrns ap- 
propriate' to the measuretncut t)f niich calegory of oljjc«(!!ivc8, 

Amor^'the outstanding features of thiti conf^'ptualixatioji tif cujniplex 
niental processes are, first. Its cinphasis .ofi (^hnjigcs iri pupil beliavior. 
It is hoped that this will exercise a dcsirnhh' inlluencr on i^ducatorsjin 
terms of ericonraging them to IVirniulalc* iheir olijcclives in more* be- 
havjoral and ineasurable terms. Stu^ond, it is Hnggt*slivc of \w\v research 
in ntnalyzing arid measuring sotne of the siiggt'slrcr calcgoricH, Thinljt 
should facilitate connnunication among edncaiors ami t(;si(*rs. 

F shall inerition only briefly th«* sccontl din^Mion in wlm^h I bt'lievp 
progress will be made in improving our undcrHhuicling of (he complex' 
niental processes^ that is lhf/{lir^Cby of HpccMfying more carefully the 
elements iti the criterion which may iWcct (he pn^dlclive val^lity of 
any particular application of a testing device. I sually the eriieriori 
measures which we have availably are complex In that they reflect 
many elements 

One example of eflbrta to (letc^niine the f'ompOTienls of conipjex 
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criteria is the application of factor-ahalyBis to correlation matrices 
j^hioh include •both criterion measures and a /variety of factored tests., 
The loadings of a criterion 'measure on the pHtafnod factors yield ust^u^ 
information opncerning the nature«of the critprion^_ , • 

A srtqnd e^mple of effort'sJtc?,determine the components of complex 
criteria is founcl in the work of Stern, Stejn and Bloom as ekborated 
in their book "Methods in *Personality Assessment" (5). The methodb - 
described have implications for both the intellective and non-intellective 
aspects of personality. I hope that our naxt speaker will okborate oh 
these riiethofeiui;thCT. , ' 
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Wlieti I first saw the^litle qF the talk tlmt wasMis^gHtHl \o ^ne— ^ ^ 
* ' C ii1X'r|a of ^ N wii n te^k^ct Ufil A gpeot s of Persp i fa li t ' ' 1 w ^ m ^mc i n a t ed £ V^^' " * 
l)y thoyprete-^*noH^\qi the^wori '*iioiaintellectiial/' It reniindecj'me qf ^' ' 

' aii bW^folk-tale of how'the first census wai takcit. ^Eiiiy^loiig years ago, ' ■. >-;^ 
/the sj^ry has it * peop|B..iecarne interested in learning how nuiny of them ( ' i\ 
^t4iera were.- They Hned.tlieiti selves up^ appointed a ptnsus takiir^.and he I. 
coupted as follow^: ^7iot-o^iej*^ "'not-two/^ '*not=threUj" and so on. ThisP 

^ euriou^ c&nting procedure obviously Jiad a purpose, Tli^* people of the 
p^^riod were quite superstitious cind tJiey fearaGt^ that if they couiited in ^ 

' a straightforward fasliion-*=**one/' "two," "three," then tlie' devil 
would know how many p^»ple, there were and" hej^vvould be ,awa^ of 
ho>v many souls he still had to d<'9troy. NeveAltfeles^^ the people warited 
to know liDw many of them there "wei:e and, to fool thtv devil, the^ 
coun t ed * * not hdi le , * ' " " not-t Wo , ' " * ' n o t- 1 h r ee / ' 

k' As you might havt inferred from the folk-tale, I believe that per- 
sonality factors liave not received the attention they- deserve in the field 
of measurement,. Indeed,' in so^e qiiyterSp they have boen relegated to 
second-class citizenship beta use they are non intellectual factors. Some 
people shy away from tliis fltdd because they ^belie^^tl/at it is all con- 
fused. And when I talk to these people I often wonderMvhat their J 
positions would haye been if they, rather t^ran Hi net, had been, ysked 
to nieaifiJ^ iMtelligence..IIo|v would tliey^hsiW^defined ii^»IligenceP liow 
woiilij they ptoceed with this probfcrn wWmi is really no niore'nor less ^ 
complex than the measurement of personality factors. Thcr^ are other 

» people who- have beconie so impressed .vvith the fadures imported in 
certain iarge-Bcale assessment studies that not only are they ready to 
throw out assessment but also the tecliHkpjes tluft have been used, even 
though tl|e tftchniques were not really puKAo test in these studies. 

The fidd of mqasnrCThent mi\ no longer 5fiCor3 to hide behind rntk^n- > 

^ alizations or behind prolessional bias. Measurc^if intellectual functions 
^have not given all that we desire in terrns^^ understanding aiid 
predicting behavior* If you wish, the devil will h^e his due. It is no 
longer a question of whether or not one likes or Nvants to study per- ' 
ionaUty factors. They must be studied. It is no longer a question of 
whether[the problems in this arj^a are difficult. They are. ' 
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In the yme allotl^ me I shall assnrne tluit we are well ngrecd 4h tu 

the importance of studying^personaliLy factors and I sIinlHiurn to a 

consideration (^onie'^a^dr problenm that arn cnticd obslnrli'H in tlip 

way of proglrcBs/ " * ^ . ' 

. - ^ _ ■ ^ * . 

V ^ * , Vrtterion * , ' t ' = 

What isja critorionlVTheru aru two*Jnnjor''\vayH in whhih this tm'm 
has bpcA'uiod^nd it is iieccssary to dilTcronliHte butwtjen tlioin for 
puiposes of ouK discuBsicHh^ 
M^.A critcrioi/is a terin assigned U| a iAnm of an incHvidnurH be- 
Jiavio^or responkm. This T dinll rulbr !o lafer an '^vtkm (•mterion." 
2, il uriterion \% tliu berni a^i^i^crTcjt a slaiKlarcl ul' pcM-fcjrrnanco^' 
' . This I ghall fcTer to later as^'i)eV(<jrinaii(/n c'riuninn/' • - 

III- the first, '*-ulasji eritorioii,'' th(j/priinary^)urposo is 'to mtij^atc 
<J|isses or groups of^ndividuaU and iipthu ^ceund, "periurriianir|cri= 
terion," the primary i)iirpose in to stiulH^ie n^lationNhip belweeri^hcj, 
basis for separathiglgronps of individnhls and some judgment or evalu- 
ation^ The first is related to construct validity and the second is related 
' to concurrent and pnedfctivo Vnliclily. Itr tlu^ first, a typical* question 
is, "How can one syparatc persc^nn in lernm of anxiety, ego strength, 
need achicwementr^eeid ddminflnen*, etc.?" In Xhe secot^d, a typical 
question is, "Wliat is the relationship be! ween^^uixiety and good per- 
formance in a proyenvsolving situation, good perforrnance As a clinical 
, psycholQgist, good perfcrmance as atl execnUm and good perfomiance 
as a coi^pany employed, i^tcP" Since sotri6 (Mlhe probk^niH Involved in 
elap criteria are diiferent frtan thost-involved in ^erforniance criteria^ 
r shall discuss eacli^of I hem sepjirately. , 



In losing over personality tests in tertns of class. (u iteda, there arc 
. two types of problems. One typc^ orj)rohleni is ifiherent in the dinicnity 
of-the task and the pther type of problern seeniH.^to Ik* generated by 
psychologists, themselves and in in th(* natU4'o of a nonunnnieation 
¥ difllc^pty. Both these prdblerns Hverlap a ^reat deal, as you will obsery^ 
in the discussion that follows. ' 

due t)f the biggest problbrns in the ineasurerutMit field is that psy- 
chologists have not yet agreed u\nn\ a Isixononiy of personality charac= 
teristics. (Hopeftdly; we might follosv BlAonrs work as Dr, Wilson Has 
suggested.) There are, some who concern Jjiemsclves with phenotypic 
characteristics wliile otUuis concern themselves with genotypic chariic- 
teristics. One investigator considers "leadership" a personality charac- 
teristic while another considers *Meadnrship" tln^ ^oupoine between an 
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iiidividi^rs,yiecds, etc. and the! specific* situation ffi which he finds ^ 
hirriif^. This is one type of confusion. Another type of conlfusion arises 
when several invest Igatorrf usp the sanie term for a persgnalily factor 
but dWneAt d^ Both McClelland and Edwards have tests in^ 

wfiigh^i)jeea^^Wfty$fii^ is a' variable but both investigators define need 
achw^fTOtpt^n^^^fier^ifc ways. Low correlaCions between the two tests 
^'agi^@5t^^rt^ M diLrerent teehniques that are used to get 

B^^^pfiievpi^fet but|also because different types of need achieve- 
' n I l4i t jfef QrB t ii&#d a^ si^x^s ted by B e n d i g ( 1 ) w^ h o f o u nd a cqrrda t i on 
of .11 betweeir Kdwards' and McClelkuid's tests, .^hese are two" types 
of- cfenfusions t.liat psychofogisfs thomselveH are responsible for, Tlie 
dif!icultiog do^not lie in tlie 'persoiiulity Tnclors studlefi.but in the people 
s tud y inj (1 Mi\ . To ex a gge ra t e t h c (*a so a hi t „ psy r^bl ogis ts h a ve c rea t ed 
their owm Tower of BabcL /?t times it appears a^ if what is necessary 
is the creation of a technical senunitic^ HLu^iety luid^lhe auspices of the 
American Psyciiologierri Association or^sonieMjther' grdup that would 
be charged with the responsibility ibr providing us with a standard 
definition of terms. As each new iiivcsiigattjr came alon^ and selected j 
^ variable for study lie would accept tbr^ stnudard definition oi' else 
show good cause why it should* be altcjcd. 

Developing and accepting i\ tj\xononiy of pcrsDiuility eharacteristics 
would' have an'additional a(fvaii^^fe^M would lielp niake us aware of 
the gaps ii; bur knowledge. It ' \%(jula ^alurt us to those personality fac- 
tors that require more stn(iy niicr kiM^p us frfiui being redundaiit in 
our researcli. 

A second type of prob^^n n^sidcs inorc in the nature of the task but 
liere too psychologists have ndded to th(*' (M)nfnsion. Assumiiig that an 
invesligatoi' has svlc<'tcd a ptTsonality nictor liir study, he is then con- 
froiited with a rather larg<* nniscrsc of -bi^haviorj^ and responses from 
wliiclj lie tnay wisli to si* let I a Hamplc tliat would be appropriate to the 
selected construct. In llm area of anxiety, for (*xample, the iiivestigator 
is confronted w^itli the followinii possibilities; physical symptoms, func- 
tioning uridor experimentnl HtresH^b(*finvior in HO(!ial situations, defense* 
mechanisms and th<' like, Iu)r egTH^f^iTgTr^ Murray and Kluckhphn 
have listed the following three rn^yor anm^^^^rception and apperception. 
iTtfellection and conation and their fifteeii^ub-areas— exterual objccs 
tiyity, internal objectivity, long apperceptive span, concentration (di- 
rectionality), conjufietivily of tlioiiglit and speiK^h, referentiality of 
tfiought and apcech, svill power, conjunctivity of action, resolution of 
conflict, selection of impulses, selection of social pressures and in- 
fluences, initiative and aelf-sufliciency, responsibility for collective ac- 
tion, adherence to resolutions andi. agreements, and absence of patho= 



23 



195? Invilalional Conference on. Testing Problems 



\ 



lo^lical symptOHiB (2). Confrbnlud with ^lU UiCBO possibilities^ no siiiglu 
inveslif^ator can/at any one lime, study each major eonecgt as Uioroughly 
uj? it should be studied. Each investigator obviously selects thut be^ 
hnvioral constellation which interests jihn and then invesligates it as / 
thorojighly JisTOSsibie. To be sure -pVobiems are enconntered. On a Ibrmal 
level! however, they are no more severe than the problenis iirvolved in . 
uehievenieut testing^n aehievemcint testing the oxainlner eaivcs out 
a Bpeeific urea for InJi study, e.g., 4th grade aritlnnelic. ile d^^ot 
investigate the child's ^iHty in uiuthenuUics. In the personality area, 
iiow^ever, thei'fiiare relatively lew fustaures where a uarrow^ field is 
studied. P^sonallty people tend to strive for the broader (H)neepls and 
^ tlie global^ernis. Here is where psyclurlogist*? add to the confusioiu 
After they have developfc^ test or a teehnitpie fur studyiiig a spueilh^ 
aspect of behavior, they ^lle tjieir tests with global terms and thereby 
^add luueh surplus meaning. Once again consider the problem of anxit^y. 
There niust be any jminber of tests that purport to measure anxiety' 
but wbatkind of anxiety are they uieasuringP Are Ihey measuring frer 
ilbating^nkiely, bound anxiety\ gnill anxiety, shame anxiety, or whnti^ 
A recent faj^tor analysis of the Taylor Heale (3) indieatc'S tliat it muasards 
at least fiveudinbrent kinds of anxiety. ^ 

This is tie natural course of development with uftist |my test, whet her 
it concerns itself with cognitive or piirsonalUy characteristics. A/ter the 
originator of the test has presented Ins results, otluM s help by clarifying 
its purposils and areas of greattfet i'ltlcMt^y, But what also seems to. 
Jia|3pen is that a tost g(^s fixed wittf its original title and the unsuspect- 
ing investigator who ](vants a test of anxiety picks tip the one that is 
mostTeadily avnilable or the one that is most featured in the recent 
journals and uses it without investigating the type of anxiety the test 
supposedly measures and wjuUher that typt; of anxiety is relevant to 
his purposes. Indeed, this is a marketing problem and" we can always 
say ''Ccimil emplor,'^ but I think we haye to assume more responsibility 
than that. The Technical Ueconuneudatlons of Jhe A. l\ A. committer! 
may hplp pmmeasurably in this l egtud- The oifly thing 1 would have U) 
add tg lhi3se reeommondations in the Hght of what I have just said is 
that each test in.tlie personality area bV' entitlecl by both its class and 
sub-class terms. Tluis, in the area of nn:^iety, no test would be published 
simply bunder the beatling "Anxiety" but under Itie lieading ^'Anxiety— 
Physical Symptoms*'; *^Anxiety—(iuilt;^ "'Anxiety- Shauie." In this 
fashion not only would we clarify what we are talking about but we 
' would als(i curtail some of the confusion that (irises when cross=validatiqn 
is atteoip x'd. 

A third type of pro^ni in tlu^ fic^ltl of personality uieasurement is 
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^ the jq^uestion of what type of technique sliould be used in preseiitiiig tne 
^te^t stimuli to the subject. At the present time a largo vnriety ofttecI>- # - ^ 
c%/*n|qiies do exiit. There are paper* and pencil UMs, sentence complelion 
' t^sts, story tests, ink-biqt tegtSj drawing testa, siturftional tests, experi- 
'-.mentgl measures, physiolpgical measures^ etc. ^Indeed, psycliQ^gisfs 
^ ' ' have shown ingenuity in teclinique development. Since nian is a complex 
and multi^facete j orgahism.who Junctions on at least two levelsr— t^ii= 
^ ^ 4^ious an^ unconscious— suclj developuieiMs artf necessary and no doubt 
yalyable. But tiie problem, is that we do not understruid tlie lelatiunsliip 
between these techniques and relatively little tiiA? and energy is devoted 
to understanding th^se relationships. Here, I would slate the problem 
fis follows I Since rna'i> is a coniplex organism hpw dp the character of 
tlie data obtained from him w^itli one met hod comparxrwith the character 
of the data obtained from In in with another method. In some instances^ 
there may be high positive relationships, zerp relationships, or even ' 
negative relationships, Orie should nut bl* either encou/n)ged or dis- 
couraged by the magnitude of the relationship but rather b# curious 
about why the relationsliip has oeeurred at alL Typically, however, if a 
negative relutionship is found but wee n two personality tests it is more 
like I J tlian. not tliat one of thern gets thrown (iut and the reasons for | 
the negative relalionHlnp are not at alWnVestigated. To take an extreme^ 
analogy, it is as if the invt^stigator were the iirst person ever to study 
the oxygen content of the blood and, iffter driiwing two lilood samples — 
^ one from the veins and th(' other trorn the arteries — he throws up his 
hands in despaii: because he has Ibntul a negative correlation. 

There pre at least tw(.i reasons wl^ the problem I have posecl has not 
received the attentiun it deserves, imd botii of thein/have notbiiig tu 
do with the criterion problein" hut with psyehologists. First, the profes- 
sionalisni in our field has resulted in the development of specialists. 
There are Rorschacirers, TATers, paper and pencil testers and the like. 
These specialists even have their own societies and they have achieved 
a level of trained iiicapacity so tlial they find it dinicult to eornmuriieate 
with their colleagues^ Curisequently, the focus of attention is no longer 
on the problems in the field of ya^onality, but on techniques^of obtaining 
personality test data and scrffes. The second reason why this problem 
lias not received more attention is that psychologists have tackled tlie 
prediction probleni .even before they coped witli the understanding * 
problem and, when Tlie predictions failed, they were all too ready to 
discard the tests. The Rorschacli and -other projective tests have been 
used in a number of predictions and assessment studies and why they 
were used with the expectation that they wuidd be excellent predictors 
when we really did not have a complete grasp of the factors involved 

25 



19^7 rrwitalional Confehnve on Tedinii Problems 

ir^ them, is rather diflRcult to lallmm. Aftej- flio lirSl ruw i^vportcuict'H, 
there should havu'bccn suflicii^nt^vidence m lo i\w dilTlcnltii^s invfilvod 
but the experiences ware pfertmttcd to ac(Himu1ntc. J would^iiKsest that 
we brmg problemB back into Ibcuy nneLtlie teehrmiuus will" take vnro 
of themsetVes or they will be developed as the accusioii deirmiids, 

A Iburth problem iii this arenas the papulatioii Hulq(^lod forkudN. 
Let us aBSunie that a researcher desires to investigalc tlu' rolaHonHhip 
between personality factor ''x" and soinelhing oIbo called "y'\ile avouUI 
obviously do well to have some a priori basis (or asHuniing ihiil faeloi^ 
''x'\ Exists in his population and that it exists wilfi snnirieHt range to 
make the study. worthwhile/This scenis like anobvious protMi'dure J)ut 
there arc times when the investigator studies an available populatiotF, 
usually the , ubic[Uitous college so'phonmrc, oii the assumption Ufat 
persDnality ftictorsare normally distributed. Then, when Dfe invesligales 
the relationships between factor '*x^^ and "y'^ he uither lintls iicgalivl^ 
results or he docs not replicate the work uf uthcrs. Much time and ell^jrl: 
have been saved if an a priori analysiH or even a pilot study had 
undertakeiu Lest you mistuiderslarKk ike difliculty here is not one 
of experimental design or the value of n pilot sindy but, rather, a way 
of thinking! 

The limitations of time do nut allow mv to d<> nxuvv than Inuit in u 
rather cursory fashion one of llie issia^s involved. In nmch pcrsonaHty 
research we uoncgrn ourselves with Ww relnlionship Ih'Iwccmi single 
personality. vanafo and some other vuriabh^N. Ilie f^u t of the matter 
is that the human organism is not inach* n|) of iiulepcMidenl variables 
but -of interdependent variables. For e\am|)le. lo talu**Li single ease, the 
indivicliial who scores high on a test nf "aiixiei v free lloaliiig" and high 

,pn a test of "effo-strength- conjnnetivity^' is (litiVreiit fnau (uie who 
scores high on the t(^st t)f anxietit' but low. in c^*^<HstrengiiL If W(* timit 
(jurselves dmply to tlu' anxi(^ly, oin\experim(Mitnl resuKs may Ix* (luilt^ 
Hmited. Restating tins issue in other terins. it may hv said thai one of 
the difliculties in working with non-inlelleclual nu^tors as well as with 
the intellectual ones Is that we liave mA deall adecpjatrOy with the 
*'organismic'' and typology pn>ble!ns.. Indeed, these are dinicidt proh= 
lems but these difTieuliies shonid not (hMer us frni!i seeking a solution/. 
The laet that we hav(^ been dissntislliMl %silii previons all<Mnpts at typo= 

* logical systems has deterred us as has t l^^tMielief that iril peopK^ are e(|ual 
in terms of personality. Probiddy solution of the problcMn will be more 
feasible now that teehnicnics such as latt»nt structure analysis are avidU 
able. But the technique itself can oidy cuniplemenl some good theorizing. 

The lour problems I have just discussed are not ihe oidy problems in 
dealing with class criteria but an nnalysis of tluMn leads u\r la \\\v 
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uoucluilon that there is nor"criterion'' problein but rathor a piobluiii > 
< in devdloping order out of disorder. This goal will no doubt be achieved 
if we^ declare a moratorium on the intprnedne conflicts bej^een pay': 
chologists and become more probleni'Oriented, ^ 

' \-, 1 g Perjopmame CrUefia^-^^ ^ ' / . 

In llie tirnc that reniaing, 1 should like to say lew words about 
performance criteria. ^ 

Ai psychologists turn to the prediction of behavior in eomplex social v 
situations, they arc often discouraged, by the fact that sorne large-scale^ 
assessrAent studies have failed. It Js often suggested that these studies 
failed because of difflaulties involved in , the ^'criterion probleni." I would 
like to BUggest that tliese studies failed not because of the ■'criterion 
problem'' but because the researchurs did iiat dilTcrentiate between a 
criterion aiid a standard of performance. When the assessors tried to 
pick the ''good clinical ps^Qjiologik," the ''good researcher,'^ or the . 
^7*goDd piychotherapirt7'^ they overlooked the significance of the word 
'*!*good'' and tlie fact that people were making these judgments. In other 
words* the assessors were not studying ''tlw good clinician"— they were 
studying men who wqyq' regarded as '*good'^ oc,/*poor" by a group of 
significant others. The judges, by their decisions, had signiDed who liad ^ 
and who had hot attahied a certain level of competeiice.^TAese standards 
1 of performaiiae are not psychologiual variables; they are social evalua- 
tions. The psychological variables or wliat I would call the criteria, are 
tliose which tlie assessors assuuie to be involved iii fulfilling the demands 
of the, situation. It is the assessors task to '*get behind" the judges' 
verbal sliitements and determine their bases for evaluation. Having 
ddne \his, he^ay then be able to decide on the psychological variables 
he will itivestigate and the instrumentB or techniques he will use to 
obtairi liis^data.. 

I can perhaps make my point a little more clear by citing an experi- 
ence during the course of the research that resulted in the book Melhods - 
in Personalily Assessment, by Stern, Bloom and myself (4), . - 

We had been asked by the Dean of a school to select the creative 
student in his area. During the course of some preliminary di&cussions, 
the Dean was asked to describe the creative student. He started by 
sayings "He has a crew cut, tweed jacket, flannel trousers and saddlft 
shoes/* The Dean had ranked his ten students in term$ of creativity r 
and the assessment group studied these students with a variety of 
psychological tests. The assessors' rank order of the students matched 
perf^tly with the rank order ;obtained from the Dean. The Dean was 
overjoyed that he now had a series of techniques on which tp base his 
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future Judgmlbfe^o be sure, the techniques were avairable but he did 
. not have; what hag been called tKe criterion. The crUerion, in many 

respects, was a very simple one. After the Dean had described the 
- creative studeht for ua and since there was sueh a close match between 
J his description of the student and. hi's own appearance, it becrnne ap- 

: parent to the assessore that the creative student f - 

be one who was created in his own image and, therefore, one major issue 
for the asseisors was to determine what kind of a person the Dean was 
^and who could get along with hirn. 

This example mustrates what I mean by gctthig. behind the verbali-^ 
zation of the judgos. The problem here is not a simple one, to be sure, 
' r but it can often be resolved if tlie psyclio|ogist spends time In carefully 
observing his subjects while, at work or in extended discussions with 
the judges. The psychologist's task is also facilitated ifjie makes use of 
methods and techniques used in the other social sciences. For example, 
in my researeh on creativity, we liave fpund it quite valuable to analyze 
^ the environments of our researchers In terms of foles and fbf two of 

• . these, the scientific qnd .professional roles, we were able to draw quite 

' ^ heavily on the analyses already available from tlie sociologists. 

* In Methods of Personaliiy Amesmienl we have also suggested the Value 

of developing the models of personality necessary to fulfill the environ- 
^ mental demands and we have alsq indicated seveK^ approaches for 
arriving at and clarifying these models. 

Comparing the discussibn of clam criteria with the discussion of per- 
fbrmance criteria, it is apparent that the role of the psychologist in 
both situations may in ccmin respOcls bo quite different. In general, 
when dealing with performance criteria, tlie classes of subjects are de- 
termined for him by the judges and more often than not they are not 
psychologists. When dealing with class criteria, the psychologist genei^ 
ally determines his own classes or works with sonu'orie who has been 
trained in this area. Some psychologists often find it difficult to accept 
the role demands in the assessme!it situation. That is, when tlie psy- 
chqlogist is asked to select tlio "good clinician," ''good leader,'' etc., 
he often has ideas of his own as to what such atperson should be and he 
is likely to select these variables for testing. But, unless his thoughts 
^ are congruent with those timt exist in the minds of the person making 
the judgments, he is likely to ftiil. In the assessment situation he is best 
off if he is quite realistic and deterifiines the reqnireinenls of the held 
before he selects his variables and techniques. 
. In concluding .this section on the critorion as a standard of perforin- 

ance, I can only r^eal Lewin's formula DPE— Behavior is a function 
o£^e transactional relationship between ihe person and his environ- 
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^ent. Although we still have much to learn about the person, it is 
incumbent updh ua also to understand the environment or else our 
predictions will fail,^ . ' '. ' f ■. 

In summary then I have presented a ieries of problems that confront 
us when we deal with criteria. These problems are e^entially not very 
diff^ettt whe^er we colder the intellectual or nonintellectuEd aspects 
of personality. While I may have raised more issues than I have an= 
swered, I cannot help but feel that they will be resolved ae we devote 

more time to analyzing them and to clarifying the basic issues. 

^ * -• . ' = 
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PaHfctpante; Amje Anastasi, William E. Coffnian, Joshua A. Fisliniaiu 
Leo Nedelaky, . Morris I. Stein, 'Robert C. Wllsoih 

- ^ V • - ^ ^ 

/ DR, NEDEtSKYi Dr, Stein believes that a psychologiit s descnp^ 
tion of his ow^ work as a study of aiixiety is not precise enough ; the 
descriptiQii shpiUd indicate the type of anxiety studied^ e.g., anxiety- 
shame or anxie^^guilt. I don't know whether piychology has reached 
the stage of development at which r^iental coiistructi ^uch as anxiety^ 
sharae—and I take it that the concept is a merital construct and not a 
directly observable phenomehon— are so soundly based and firmly 
Established that a^^ psychologists must use them as terms of analysis. 
In physics, there was a time, a very productive time indeed, when many 
key terms and concepts were fluid and equivocal. The seventeenth and 
eighteenth century physicists in their search for concepts that are basic 
did not agree on the meaning of mass, heat, quantity of motion, force. 
Instead, they went their separate ways and used these terms tentatively, 
e,g., Huygens used mass and size interchangeably, and, as tools of 
analysis, in a way that seemed most fruitful to each investigator. It was 
hot' till physics was considerably* more advanced, that the meaning of 
these concepts was crystallized and set. In contrast, some centuries 
earlier,, there was a set of well-agreed on mental constructs^ — ^the four 
elements: earth, water, air, and fire, of which all things were thought^o 
be made. I can imagine a paper presented at the University of Paris 
in the Middle Ages being turned down for "lack of clarity and precisioif ' 
because it described a new subitance without specifying tlie proportion 
of the four elements of whicli it was composed. 

CHAIRMAN FISHMAN: I wonder if either of tlie speakers would 
care to react to this? ' * 

DR. STEIN: Dr, N^delsky has certainly poinLed to an important 
problem. The scientist s language and his ciassification system may 
stand in the way of progress. Unless the scientist takes time out to 
question and test his classification system, he will limit himself only 
to those data for which he hai a structure. In my talk, I tried to make 
a plea for order, not constricting order, 

tSlR^'ANASTASI: I, too, should like to direct a question to Dr. Stein.^ 

It IE my impression that one of the principal objeotips of factor 
- - : ^ ; ,^ - 
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aimlyais ii taxQtiomy. I don*t mean to imply that all factor analysti. 
have been successful in this regardi but I am wondering how Dr. Stein 
feels about that tnethodology, and how he feols about it in coniparison 
with%ther inethodolc^i^ in arriving at taxonomy, - 

DR. STEIN: I have high regard*for factor analysis. I fe&r,vhowever, 
that we may have permitted our statistica|( tecfiniques to affect and 
direct our thinking beyond what is desirable. I do not phject to i tat is- 
tical techniquea but to psychologists who use them iustead of thinking, 
Rather than sit down and theorize about personality or tlie personality 
factors involved in a research^ some psychologists tend to jump to 
writing items (because it is easy , to do so) and thori use statistics to, find 
out whaj they mean. Tliis can result in empirical relationships that may 
be difficult if not impossible to explain. I guess that tin's is a statement 
of faith that I do believe that the problem of^rusearch should first be 
analyzed in terms of the personality factors involved, then the appropri- 
ate test$0r items^ selected, and finally, the results subjected 
to statistical analysis, , 

DR; COFFMAN: I would like to ask Dr/ Wilson a question. 
' You indicated that in identifying the talented, you were concerntkl 
with other characteristics than the inlWligcncc as measured by com- 
monly used tests: Have you made any study? Have you given any 
attention to what I presume is a group of youngsters in tliis situation 
who are not identifled^by your various methods of defining talent, but 
who, made very' high scores on-intelligence tests? 

DR. WILSON: Do we have students who score high on intelligence 
tests but do not score high on any of our talent devices? We have not 
singled out*^ch students for particular stydy. Most of these talents 
that I have Mentioned are in the various arts* We find that they are 
somewhat independent of standardized tests of intelligence in that there 
are many students, who are quite taleiTted in tlieso special talents, who 
> may not be very bright. " 

CHAIRMAN FISHMAN: The question was referring to the reverse 
situation. Do you want to follow it up a Iittlc*further? 

DR. COFFMAN: Yes, I was just thiitking whether tun years fioni 
now somebody might not be wondering whetliei Ihby had missrf a bet 
in not paying some attention to these bright, ''untalented'' youngsters. 

pR= WILSON : I don't even know ifthere are such, or what perceutage 
of the populaticfti they constitiite— I expect a rather small one. I didn't 
mean to give the irfipression that we are only interested in the special 
talenterWe do, 0/ cowpfaj identify students on the basis of tests of ini 
telligence and scholastic aptitude and the major part of our progran^ 
is roncerned with uiaking better educational provisions foduch stu- 
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dents, I emphasizi^ the creative areas in this talk since tlie^^ are the 
areas which are less commonly de^lt with in programs for the gifted. 

CHAIRMAN FISHMAN: This is really a very serious problom frorp 
any point of view— the. point of view of just humanity, or point of view 
of natural resources-rtodividuals of very superior ability whose talents 
we can not discoverV Tlftre is a rather serious area for research. 

UNIDENTIFIED SPEAKERi Question for Dr. Stein. 

You were referring to the disappointing results Obtained in very 
particular studies using the tests against the usual criteria we have, 
and you suggested that the reason for this might be because we have 
too little understanding abopt the pature of ijian. 

I would like .to suggest for your cpnsideration that one of the ways 
we have of achieving that is to get some measure of relationship among 
our criteria and reference variables. This k an approach which in the 
past has yielded some useful purpose. And ^^el that while a test Svhicji^ 
shows diMippointing results against. some sort of criteria stiouldfi't . 
necessarily be the reason for sorrow,: neither Should it be the r^son 
for rejoicing* , ^. / 

^ DR, STEIN "The point I tried to make was that if we really j ^arified 
ouf criteria, we might have more occasions for rejoicing. To segard 
"good student'' or "good chnician'' as the criterion without under- 
standing the psychological factors inveJved in arriving at the "good'' 
evaluation will certnirily limit ns in our atteniptB to make vnlid pre- 
dictions. 

QHAIRMAN FISHMAN: Our time draws to a cloRe. I think you 
will agree with me that this just scratched the snrfaee of what might 
be involved in this area. Certainly, both of our speakers Iiave empha- 
sized more the conceptual angle (which nught fneiude the semantic and 
the various communication problenis) in refining our thinking in this 
area, rather than any of the otiier pOHsible approaches to criteria, more 
ultimate or more behavioral approaches to wliat we liave in mind with 
our tests. But tin's is the kind of topic we are gtnng to have to return 
to many times. 
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J Remarks of the Gliairmari 

' l.^WAYNE WRrGHTSTONE 

- - - ^ - : _ -■ '^^ . .-' ;-• - ' - - ^- ^ 

Althpugh the words ^-exercise writing' ' in tht title of the second session, 
^^Improving Measurement Through Better Exercise Writing/' are some- 
^at ambiguouSt most cf us here today, I am sure, reah'ze that exercise 
writing is intended to mean the formulation of objective test exercises* 

'We are thinking in much broader .terms than in item writing because 
we are concerned with the totaf situation on which the items are based, 
as well as the items themaelves. 

It ji our hope that the type of exercises that wifl be discussed at this 
session .will f be the kind 4e9ignfed to tap some of the more complex 
mental processes rither than recall or recognitiou of facts alone. More- 
over, we hope that student answers, or responses, can be recorded so 

""that^ scoring n^ll be objective rather than disay examination type 
answers, which are frequently assoclateH with the term "exercise 
writing,*' and require more subjective scoring procedures. 
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I am indebted to the next ipeaker, Lto Nedelsky. a former colleague 
^1 the OiDce^of the Univereity Examiner at the University of Chicago, 
for an illuitration of a clasric type o£ test^exerciifi in the humanities. 
When he graduated from high school as a refugee from Russia after the 
Revolution, there were only %0M0 such graduates in all Russia ;^ence 
the fmal exafninfition was quite an dccasion. Candidates had to appear 
in a pi&Hc' place several' hours a day for a whole week and answer 
any questions that the townspeopre wanted to ask. Leo was'almogt 
caught by^ the Archbishop of the Orthodox Church, a kindly old man 
ythen in y § ninety'third year. After the examination had proceeded for 
/some time, the Archbishop indicated that he would like' to ask the young 
■^i^iag^l^i. '*Y6ung man,'' he said, ''when the soldiers opened 
C^ri^K^^h the lance, why did both water ai]d blood come forth?'' 
Leo wts stuck; he was not prepared for this question. But as he put his 
head down, and appeared to be tliinking furiously, it occurred to him 
thQt the old man was really in his dotage, and if he could only^postpone 
his reply for about a minute, tlie Archbishop might forget what he had 
asked. He therel|ite continued thinking, and after a suitable pause, 
raised his head rfS^though inspiration had struck and replied, "Your 
Reverence, when the soldiers opened Christ's side with the l^nce, both 
water and blood came forth!'' 'That's right," said the Archbishop_ 
benignly: ''A prize pupill" Leo said that, as he glanced around the^ 
table,, he saw a black-bearded Jesuit gazing intently at him as thouglT 
to say, **We need that boy in our Order!" It is something to be said for 
this form of, examination, that, aitjjough Leo niight hav.e had theological 
objections, that Jesuit was undoubtedly right. . 

For all its merits, this form of test-exercise has all ^t disappeared 
except in the archaic ceremonial of the oral examination for the Ph.D. 
Coming down closer to bur own time, the sort of test-exefcise that is 
still most commonly used in the humanities is well illustrated by a story 
that my old professor of the Bible at Harvard, Kirsopp Lake, told in 
one of his last lectures. It was the day before our final examinationfand 
r think he tried to ease the tension by ^ying, '^Gentlemen, I had a 
- remarktble dream l|st night.^ I dreamed fiiat I was sitting oaXcloud 
at Judgment Day, watching all the tribes of earth assemble. They all 
came; together into a great plain and sat <down. Then, out of the cir- 



36 



35 



ERIC 



"camambient mist, a peat hand arose and began writing on a celestial 
blackboard in lettirs large enough for all the world to read. It wrote out 
the -Ten Commandmehts^ and then— in typical examination fashion^ — 
it added- 'Stwdents choose six/ 

two types of questions pretty well cover the history of examin- 
ihg techni^M in^e humamties down to our own time. To carry on 
the story from that point, robviously have an enonnous field to cover in ' 
a very brief time. It usual|y takes me a semester. With your permission, 
therefore, I rfiall pass over the fields of art and music with only a brief 
qqmment on each, cbiefly because they require so much testing time per 
item before I cbiild make them come alive for you. I should have to 
have at least tape' recordinp and slides, and I prefer a Uve pianist and 
original works of art loaned from some museuin. If I had the pianist, 
it would not be difficult, but it would take b good deal of time, for her 
to play snatciies of* twenty very familiar melodies, like **My country, 
'tis of thee," and "Old Black Joe.'' In about half^ of these excerpts I 
should coach her to insert one definite but not too obvious error. You 
would mark each excerpt C if it was correct and E if it contained an 
error. If I wantQ^ to increase the difDculty, I might adt you^to indicate 
what was Vvrong with each except that you marked to mark it 1 if 
the error was in the melody, 2 if it was in the harmony, 3 if it was in the 
rhyHim, and 4 if it was in the expression or empliasis— that is^ in the 
relative loudnesB or softneas of the notes, 

If you were too .sophisticated for this rimplo exercise, I might aslrlier 
to play the little tune from Haydn on which both Handel and Brahms 
wrote variations. I should then ask her to play the basic variation by 
each composer, tell you who wrgte it, and ask yqu to tell me the chUif 
difference betw^n the two styles out of four or five suggested diffemg^s. 
Then I woulfl ha^' hlr play perhaps ten or twelve variations in a-raiidom 

Sprder: some by Handel, some by Brahms, some by neither, and some on 
the wrong tune. You can see how this sort of thing can get quite complex. 
If I started you analyzing fugues and telling me wl||| the differerit voices 
were doing, I should soon liave yoii hanging on ropes* 

If there is time, as there is in a course,'^llhough not in an exaniination, 

"T'like to make a point about any great work in sonata form by playing 
a complete sonata, quartet, or symphony and substituti^ one move- 

^ ment that does not belong: that conieS but pf a similar work by the 
same composer, written at about the samj^eriod of his development. 
One might think that when a composer writes.a work in four movements, 
-he-writes one tune for the first movement, another for the second, 
another for the third, and another for the fourth,^and then puts the four 
tunes together, with perhaps some relationship* in key or mood, I am 
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convinced that iiotiiing like tWs happens; he writes basically the same 
tune in four different v^^ys— although sometimes, I admit, it takes a 
good deal of lubtlety to recognize it as the same tune. What any sensi- 
tive rausidan can do at once, however, is to recognize it if you put m 
ti^ wrong^uet, the wrong slow movement, or the wrong finale-and 
not beoause'he has any previous acquaintanoe with the work, but simply 
because they don't belong; they won't fit. I have found this so%of • 
exercise very good for demonstrating the uiiderlying uirity of a classic 
composition in four movements. ' 

For the visual arts, one of the sitnploat tecliniquos is to use four 
projectors that can throw four slides at once on a large screen and ask 
a series of questions about them lliat can bo answerad with the number 
of the correct slide, or with "None of them." For example, at the 
humblest' level: which one was a watcrcolor.s Wliicli was nri oil? Which 
was airescoP Which wm a wash drawing? If they represent well-marked 
historical styles, which was Italian of the High Bonaissanco? Which 
was Dutch? Which was eigliteeiitli century French? Which was 
eighteenth century, English? All such works, of course, mu^ be pre- 
vioudy unknown to the students. At a slightly deeper level, one might 
show three works by one painter and one by a contemporary in a dif- 
ferent style, such m three by Cezanne and one by an Impressionist. . 
The painting that does not belong with the others will stick out, hke 
a sore thumb to a sensitive student, but others will, see no dlfTerence. 
One can ask all sorts of questions about, tuclmiques and about the com- 
position: for example, in wliicli is the basic form of the composition a 
rectangle? a pyramid? a diagonal? an S=curvo? Which mnkes the most 
obvious use of contrasting textures? Which one is basically two-dimen- 
= sional? ,Whfth is organized In a series of planus? There is hardfy any 
Umit to the number of questions one can ask in ihe form, "Which paint- 
ing does A, B, C, and D?" If the paintings have clear-cut differences,' 
and if the questions are perceptive, the answers will reveal a good deal 
of sensitivity to what is going on iti a Work of art. -J 
I should next like to turn to the ricl(l of history as it relates to the 
hamanlties, hoping to leave a clear field for Max Engelliart to deal with 
history as it relates to the social sciences. You will see the difference, I 
hope, in the kind of illustration I will use. The sort of outcome I have 
in niind haS no large social signincuiicoi it Is not a necessary ingredient 
of good citlzenshiii; it hp/mttT6c, atporso'nal significance, and is an 
ingrfedlent in one'sXEy^ophy- Lks jpompously, it is a part of that 
gebMal itbck of ideas that a man dhrries around inliis head that deter- 
raine^rhat objects and events In^^veryday life will strike him as sig- 
niflcaM, or interesting, or puzzUiig, or dangerous, or good. Among these 
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^eas I would place a hJgb value on a sense of the past and of its con- 
tinuing influence on present. *^ 

For ^aniplt, I once read in a book of popular scientific essays the 
statement ^at the average person; must take the existence of a man 
Vkf Julius Caesar on faith, or on authority* To ^ man educated in the 
.huinaniti^, that stateteeni is preposterous. I am sure that fit lemt 
two-thirds of you have in your pocket or in your purse some tangible 
e^dence that Julius Caesar exi^^e^^d left us soniethiiig that we use 
every day. It is something that you can take out and hold in your hand. 
It evenibas the n£^e, Julius, printed on It, although in an English form 
that you may not at first recognize. The place in which his name is 
printed is peculiarly appropriate In view of his life history. What is it? 

I could nudge you closer to the answer by writing the names of our 
last four naonths on a blackboard: September, October, November, 
December*^ Then I would erase the "*ber/' on the ground that it is 
notting but^^a^ have left four good Latin words: 

septem, ocloy novem, and decern. We have other forms of these same 
words in seplei^ octet, novena, and decimaL What do they meanf Ob- 
Viouslyj;gfeven, eight, nine, ten. But why do we caU our ninth month 
the seven^, our tent^^onth the eighth, and so onP Did som^ody 
lose count? That is hardly likely* We cdri get a clue as to what probably 
happened by writing out the Latin names of the months immediately 
preceding these four: Julius and Augustus men^ts, our July and August, 
Do the names JuHw Bnd Augustus ring^a bell? Certainly: Julius and 
Augustus Caesar, But how did their names happen to get into our 
calendar and displace ^11 the following months by two months? At this 
pQint I allow a-little time for Research, Someone usually look? up the 
encyclopedia' article on "^Calendar' and discovers that, when^Rilius 
Caesar was campaigning in Egypt in 48 and 47 B.C., he was n(^ too 
preoccupied with tue charms of Cleopatra to notice that the Egynjians 
had a much better calendar than the Romans. Consequently, w^tf he 
returned to Rome in 46, he brought along not only Cleopatra but an 
Egyptian astronomer named SesogineSj and with hii help worked out 
essentially the calendar we use tt)day. His successor^ Augu8tuSi<>^ecured ' 
its adoption throughout the Roman Empire, and either he or his fol- 
lowers, apparently saw to it that the names Julius and Augustus were 
^Jorever en^rined in it, disregarding the protests of the mathematicians 
that the names of the lastr four months would all be two numbers oft 
from their proper numbers. Augustus may liay^ explained that. it had * 
to be that way b^ajase Julius was born during the month we now caU 
July in^his hbnor^and by' a surgical procedure that we stiU call a 
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If I want^ to make this excise just a bit more complicated, I might 
ask you why we called this cfllndar the ''JuUaij Calendar" down to 
about the time of George Washington in EngUsh-ipeaking countries, 
and atiU liter in countries dominated by the Greek Orthodox Church, 
Kut-then^ift^ over'to a sUghtly modified calendar that we call the 
"Gregorian Calendar." One of the chief diflferences betweert them is 
that our present dates are eleven days off the oorrespdnding JuUan 
dates. If you l6ok In old history books, you will see that George Wash- 
ington was born on February 11 (Old Style) or on February 22 (New 
Style), obviously the Julian and Gregorian dates. One;of thfl Best 
exercises in semantics I know is to have a class argue about the dale 
on- which he was "really" born. If we accepted the JuUan date, it would 
make quite a difference in the time of the annttal meeting of the American 
' Association School Administratpri. Why -dicUwe change? What was 
done tojrevent losing eleven days againf Did it work? ' 

A still liiore compricated problem is tlial all our names for the montlis 
are Roman, while all our names for tlie days are Anglo-Saxon, yet eMh 

■ of our days corresponds planet for planet with the Roman nafties. SdPfc 
dies and lunae dies, day of tlie sun and day of the moon, Sunday and 
Monday, are obvious ext.nrples. Wednesday, wln-ch is Woden s day, is 
less" obvious, but his planel was Mercury, as you can see m the Latin 
Mermri dies or the Frencli Mercredi. Tliursday is obviously Tho% 
day; he waa "the thunderer." as you uun see in the German namefor 
this day. Donnerslag. But so was .fupiteutthe thunduier; hence this day 
was Jovis dies, the day. of Jove, in Latin,; or Jeudi in Frencli. AH the 

' ; other day names are Anglo-Snxoii eg'uiSiprenls of the gods or, planets 
ior whom the Romans named their dnysv^'hon do^you think the Anglo- 
. Saxons had a chance to pick up the Roman n^me4ft>^^ the ^nys and then 
. translate therii into their own jQiiguage? They did not iuyade England 
until the Roman Occupation had ended, and they h^d litHe if any con- 
' tadt with the Romans on the coiitinenl. And why.dia" * they translate 

the months? , , ,.i ■ » 

' iT.^yoju are wondering how I would put problems hke these into 
■ ^aitiEJe^boioe fbrm.^he truth is that I wouldi^t ^ther. I'd give my 
>cla^e8%MiieIingBiStio data, some historical dat#^,ajUttle time for re- 
. search' ^ask theia a seriM of questions, and let tHem answer m essay 
i^^nn; perhaps 'some preliminary dlscuBS%.- ¥ou may be dis- 
ttitibiiited btecaute you know that the grading of a A^e ^ssay is un- 
. "ieUabli a& a^ test or But I am not talking-^hout a test ; 

V i amtalkfng rfibut exerCBes for a course; and after I get siiteen to twenty 
h A such kercises over a piod of a year, and grade them ^he^ay I grade 

■ tl}em, the reliability of the cumulative total score can easily exceed .9. 
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Please note that thti program dgfefe not require me ftcchi^^^rfy. 
about fnuitiple-chig^e tes^i the tppiclii^ **B^erjCise^^^Hin^^ v' ; * 
I I mighicarfy this san^J^ 

butcome df the huirianitles that I cjall a sonsf of'the iitt^cOT of 
idefit, *eipjiCiaUy acS?ta languages ppd cultures^ Fp^r M^ I - 

learned that the newf^ranquilking drugs had beeh giVdh the;pgly riarne, 
aiarawm. J greeted ii as -an old frierilvlt was thfe rjingihg-^attle^ery* of 
my flghting days in c^ege, the watch^prd of the Epicur^ns, Aiar^ia, 
which may ba translated, * 'Do not be distutfiedl'^ Why^^had such a^ j 
appeaWor young n^en | have never^ figured out:^But I fdtip^^ tlie sanicjjf: 
idea esfprbssed in Horace's. motto: "*NU adriiirarirV^^hW^ M^ be trans% , 
iated, /"Don't be swept olf your feet!" There is ev|n a hin| of it ijo the ^ 
Biblical, OTake no thoughi for tto biOrrow,'' butth^t is really n clilTerent ^ 
idea: it means trusting^in Gcdj not irt your own irinei^ respuh^es^J^ 
later on th^ same idea is picked up iii Castigliohe's- ideal for a touriierl 
wbicl^^^' caHed iil' the sdventeerrtlti^bntijty^l^ 

ideapi^fthe honnile homme. CoQiirtg dowH to our owix time, rdhink^c ^ ; , - \ \ . 
have^e same idea on a somewhat different plan^if6?dprd re^ervgjffirui * | * , 4 ; 
Harvard |indifference. It may be something bRSic in the cpde of tli^^, -^ y'.f^ 
gentleman that has; peren^^.ppgeal for youth. Our young ihen^ mle 0 ■ J 
at their moods seem' to be grpping fbr "UJie still point pf ^ji^jsturrijn^ ^ ^ - 

worlds' ^; 1:^' ' ' ■ "-''^'^ / V " 

Here, again, I would not f tteuipt to put such an exercise into dhjjgtive 
form unless I had soriie compelling 'reason, such as having to give a test; » ; 
to 50,000 candidates. For a class it would be belter just to give a list 
of these slogans, perhapi with translations, and>aric wha|t}iey had hi, / ' / 
common^ why they have ^uch an .appeal for youth, and which oiie is ^ 
farttiest oijjt of line with the others, ^s I tbelifeve the Biblical passage tp be. 

But I hW better give sofhe attent^^ ■ ^ 

Ihafr^^rpose 1 shall qhoose a poem' bqcuuse nowhere else can I find so v * 
n^U^h jneahing in such compact forrn. jl^liis one is cailed Spring and FalU * 
and'was.written $y Gerard Manleyl^ . 
by a matire mad to a young girl, aiid h^nce the title mi|ht be taken W ^ - 
refer to ybutH and aga, I hope to ^iipnce you^^b^^ ■ , 

that this is definitely an erpor. SprLiH^ may ^Well r^f^rta youth, but /a// 
in this titla'has a|much i^re sinister me^ung tli^n '^ge/' It is a moot 
point whfether one can ask objective qfetjpiia about a poem without 
lacerating|it. r biheve that one can eliffi a very sensitive reading of a 
pora,Byj(^ective questions. In fac t, \JKen anyone ask^me whether I ^ 
rMkUy mSarstand'a very "difllcult poem, like one of Eliot^i^^par Qmrteti, 
1 have t^^Aiwer, donH know. I havfen*l yet got^arSut^ to making 
up a test: offlitl j * ^ > 
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It might be belter to begin by reading llie poern straight through, 
but it is rather puzzling and would not ''g^i across" at first reading. 
Hence.we ihaU take it by pairs of lines and ask somewhat abbreviated 
dbjectiye ^ttestions about them* \ ' ' 

'^Margaret, are you grieving ' i 
Over Gol^ngrove unleaving?" ^ 
Here are two puzzles. What iiunleaving? Staying? Failing -to j^rqduce 
leaves? Unfolding leaves from buds? Or slicdding leaves? The only one 
of these that,might cause a yonng child to grievo is shedding leave8--in 
autumn. Hence Goldengrove can hardly be a particular flower or Bhrub 
but just a patch of woods in autumn, with its characteristic golden 
color. " 

"Leaves, like the things.of man, you r 
With your fresh thoughts care for, can you?". ' 
Here the syntax is a bit tangled for the sake of the rhynie, but it 
obviously means, *'Can you, with your fresh thongte^^care for leaves, 
like the things of man?" We should get at the al^ity ^ unravel the 
syntax by a grammatical question. What would be the Qpjjosile of /^tbe 
things of man"? The things of wonianP The tilings of children^ Abstmct 
ideas? Probably it is *'the things of nature," because what shd^is^aVing ' 
for at thia time is the falling leaves, and the poet expresses some surprise 
that shf can ^re lor these things, which are certainly things of nature, 
just as she,^j|^s for the things of man. , , ' 

tr ii^j^^ as the heart grows oldur * 
It will come to such Bights colder 
By and by, nor spare a sigh 
Though worlds of wan wood lenfrncal lie." 
The puzzle here is the interesting formation, *4eafmeal," which I 
have never seen anywhere el^e. We have to figure it out by analogy. Is 
it like oatmeal, cbrnmeal, last meal, or piecemeal? The firsftthree would 
be ridiculous but the last suggests a clue. ■^Piecemeal" means 'Apiece by 
piece," Can *'leafmeal" then mean "leaf by leaf?" It certainly makes 
sense, for that is the wa>^ihe leaves would flying. The word ^*^a»wood'' 
is not in any dictionary, Britisher American, wjiich is snipir enough to 
lie on my desk, and it Js hardly worth tradiing down in anything larger 
or more specialized since it obviously: f^s to be some kind of dead 
foUage which is lying in great quantities, leaf by leaf, Upon tli,e ground. 
As Margaret grows older, she will not spare a sigh at such a sight, 
^ *'And yet you usW Weep anrf knos| why." ' \ 
As thb poem is usually printed, thrpoet has indicated a strong stress . 
on will and on and by acceiit.marks. This is a first-rate puzzl^, sipce the 
poet has just said tliat she will confe to sucli sights .colder and will not 
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Spare them so much as a sigh. If slie then breaks down and weeps, with 
no reason given for the change in mood, there is a flat contradiction. 

The only way out of it that I can sea is that she is weeping mow, *^yer 
the falUng of leaves; the poet has tried. to comfort her i>y telHng her 
that by and by she w6n't care; but slie refuses to be comforted: slie goes 
right on weeping and wants to know why she is weeping. This is the = 
volitional me pf wHlM^s in **He will do it no matter what you say.*' It 
might be translated, *'In spite of wliat .1 have been saying* you insist 
on weeping and on knowing why." I can't see any way in which it could 
be weeping in the future, for lie'has just said, that in the future she not 
only won't weep; she woirt even sigh, Now he begins to tell her why, 
*'Now no matter, child, the name: 
Sorrow's springs are the same." 
He doesn*t want to tell her just the name of what she is weeping for, 
suice now it would probably mean nothing to her; and anyway, the 
ultimate source of all sorrow is tlie same (**sorrow's springs"). 

**Nor mouth liad, no, rior mind, expressed 
What heart heard of, gliost guessed;" 
Wfe have to do a bit of translating liere. "Your heart heard of, and 
your spirit guessed, what had'not been stated in words or even formulated 
* as an idea." I should simply ask, "Which of the following is the best 
reading of Ijnes 12-13?" 

*'It h i\w blight rnan sv^s born for, 
It is Margaret you mcrf'n for/' 
Here is the ultimate answer to thcj^estion of why the young girl is 
' weeping. In the falling of the leaves she jias unconsciously, glimpsed a b 
symbol of her own death, far olT in the future: "the blight man was born 
for." It is really not the fallirig leaves shu mourns for, but for herself, 
^iv even tliougli it is only what her heart told her, and her spirit guessed; it - 
^ " had never been explained to her in words, nor even entered her mind as 
an idea, riiat is wliy sorrow's springs are tlieVsame, for tlie ultimate 
source of all sorrow is death. It is alno why the Viame is no matter, now, 
for it would mean nothing to her, but in her heart she has already 
guessed it. . .f^^'" ^ / 

If anyone now wants to iriterpret-the title, "Spring and Fall," as 
"Youth and Age," he niust have a auntimen^tal aversion to the obvious, 
interpretation, once we hav(' puK^li'd' it out, which is "Youth and 
Death"-=the latter symbolized by^ thd^failing lenves. There arc very 
few interpretations of a cryptic poem that one can absolutely rule out 
as untenable, but I believe "ago" in the title of this poem is one of 
them, iliere is no clear reference to old age anywliere in the poem. 
True, as her heart grows older, kIic will no longer weep at the falling of 
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leaves, ^ut that could well occur by the age of twelve. Thp poet la repre- 
sented as somewhat older than the girl, but there is no reason to suppose 
that he is very old; he might be a nian of thirty. The girl ii certainly 
^ not weeping about his age: **It is the blight man was born for*/It is 
Margaret you mourn for." That seems to me an unmistakable reference 
to death, 

I have now used up my allotted number of pages and have discussed 
only five topics: a few techniques for getting at pensilivity to what is 
going op in a musical composition or a_^ork of att; a few informal ways 
of reveaiing underatandings that are'' peculiarly ^characteristic of the 
humanities: namely, a sense of tlie past, and a sense of the interconhec- 
tion of ideas; and/finally, some objective means of eliciting a sensitive 
response tb a phildsophical poem. Since I would use the same types of 
items to reveal more than a superficial grasp of any prose passage, 
literary, philosophical, or historical, I hope that in some way or other 
I have touched upon the chief fields usually associated with the humani- 
ties. To attempt a comprehensive coverage of such a vast domain, so 
diCTerently con^^cived and differently taught in different institutions, 
would obviously be absurd in a brief talk. I can only hope that I have 
left you with a' few testmg ideas tliat are fnirly representative of the 
spirit of this great domain. 
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This paper is concerned with general principles to be used in writing 
test exercises, especially objective test exercises, in natural sciences. 
Most examples will be drawn from physics* the field in which the author 
feels most at home; the geperalizations which these examples are to 
illustrate, however, should be applicable to any of the natural sciences, 

Exercm Variety 

' In these relatively enUghtened times it may be safe to assume that 
the test writer has before him a list of clearly stated objectives, Le., a 
description of the kinds of knowledge and abilities the test is to measure. 
It is well, howeverj to look into theprigin of the Ust. The most important 
function of a science test, perhaps its only defensible function, is to 
predict th6 student's behavior when faced vvith a situation in which an 
understanding of science is useful and important. Such a situation, 
which we shall call a criterion' situation * may occur in the student's 
academic career * his life as a citizen, or his work as a scientist. 

One way to test the student's criterion competence, i.e., competence 
to deal with a criterion situation, is to present the student with realistic 
problemSi i.e., with situations tliat closely resemble those he is likely to 
face ^ the future. A test of this "synthetic" sort is quite valid but also 
almost prohibitively cumberspme and expensive, for genuine problems 
facing a scientist or a citizen are usually very complex ancf not of the 
• pDper-and-pencil type, Further disadvantages of such a test lie in the 
difficulty of assembhng an adequate samplu of problems and ^of com- 
municating about the precise nature of the test and students' *'Bcores." 
' Another way, is first to analyze the complex competence to solve 
, genuine problems into its constituents and then to test for the more 
important but still tractable of these. Such an "'analytic" lest has all 
the usual shortcomings of an analytic representation of complex and 
incompletely understood phenomena: neitlier are all the constituents 
known nor do they add up to the whole. In addition^ some of the more 
• important constituents^ such as habits of thought and attitudes, cannot 
be conveniently or accurately measured. Those shortcomings of an analy- 
tic test which result from the unavailabiUty of a complete analysis of 
a complex competence aye probably best alleviated by Including in 
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the test exercises that, in the aggregate, evoke in the student a great 
variety of niental processes in varied patterns. 

We are here concerned with the^ analytic type of tqst. The list of 
objectives is the list of presumed constituents of the criterion conit 
petence. As has been suggested In the hist paragraph, tlie test writer 
o4n in some respects transcend statements of objectives and come closer 
4o the criterion competence by varying test exercises as widely as the 
limits of the formally stated phjective permit. Thus, e.g., in testing for 
the ability to interpret pliysics datn the following varialions are possible. 
The data may be graphicah iminencnl, or verbal in form; the content 
may be, e.g., mechanics or heat; the student may be asked to draw 
possible conclusions, to assess the accuracy or consistency of the data 
within themselves or with other data, or to estimate the cogency of the 
data as evidence for a generalization. The forni of the test may also 
vary' essay 6t objective, 

^ Exercise, Wriiera Prepamtion 

If ifie variety of exercises described in the preceding paragraph is 
not to be merely haphazard, the exercise writer should not only kiiow 
the subject matter tested but also have some understanding of wlmt m 
involved in solving a genuine scientific problem at a level of sophistica- 
tion higher than tliat of tlie students. As will be shown below, Iv^must 
also know the extent and tlie range o£ the students' knowledge, abinties, 
and general intellig^cc. He must evep know their method of preparatiun, 
enough at least to be able to judge the degree* of novelty a partic4ilar lest 
exercise will have for them. Finally, the exereiae writer must have some 
notion of the more common nVethods used by the students in solving 
various exercises. 

Objeaiive as Bundion of (fie ETercise and of ilie Stmknis 

Having described the prerequisites for writing an exercise we shall 
skip the actual technique of producing it^ fust version and deal instead 
with the general methods of criticizing an already written exercise. The 
key question is of course what a particular exercise measures. This 
question we shall assume to be equivalent to tlie following one: If the 
exercise is successfully performed by a group of students and unsuccess- 
fully by another group, what is the main diflerence in the nbihties of 
the two groups? It scqms clear at the outset that the answer depends on 
the choice of the whole group. We shall nevertheless labor the point at 
some length because of the prevalent misconc^tion that the ability 
measured by a test is a function of the test alone. Some test makers 
seem to think, for example, that a test designed to pick^out promising 
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college material from among the studenti of an eastern jpreparalory 
school can do the same job effectively when given to the stud en Is of a 
rural school in the South, We shall illustrate our point by an example. 
. A group of students is presented with, the following exereise: 

It is possible to hear the sound of a fountain behind a brick garden 
wall although the fountain cannol be seen. This plienomenon 
A— can 
B— cannot 

be explained on iKS^^^is that long waves are diffracted more than short 
waves. \ 

Ti^e group will be divided into two subgroups: tho^ who pass by 
choosing the right response, A, ai^ those^who fail In what respect do 
these subgroups differ? That is, what relative ability or knowledge does 
the exercise test for? If the group consists of graduate students of physics 
of a Spanish University, it seerns clear tliat the two subgroups will differ 
in their ability to read English, for it may be safely assumed that all 
members of this group possess the requisite mastery of physics but that 
only some know English. Next, let us suppose that the group consists 
of students thoroughly trained in the theory of wave motion, and ac- 
customed to using tlie terms bending and spreading in place of the term 
diffraction. The two subgroups are most likely to differ in their knowl- 
edge of terminology. Let our next group consist of students who have had 
a year of physics, a good discussion of diffraction of both Ught and 
sound, but no treatment of th$ relevant similarities between theni. Let 
us assume that most of these students know that sound waves are longer 
than tliose of light. The two subgroups are most likely to differ in their 
'ability to surnflle that the principles of diffraction are applicable to the 
situation described in the exercise, an ability which may Jdc considered 
a part of the more general abiyty to relate genera Hzatioi is to specific 
situations. Let our Jast group consist of stuc^^ts to whom the particular 
* im% situation— sound of a fouiitairf behin®l wall=was explained in 
'-^dpy or textbook. The ability to recall the situation may be influenced 
by a variety of factors, especinlly if the explanation was not empliasi?^ed. 
The main difference between thoytwo subgroups is liard to determine 
and may not depend oii any tiling educationally significant. An exercise 
writer who uses situations very similar to tliose in the textbook even 
in a test of pure informatioii is oti slippery ground. 

It is sometimes argued that every pair of subgroups discussed in the 
preceding paragraphs exhibits the same important difference— the pass- 
ing group knows the answer to the question while the failing group does 
not^ — ^and that, in measuring the general mastery of physics, it is a matter 
of secondary importance how the various students arrive at the correct 
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finswer: It seems that this argumenL is hamd on at least two qtifestionable 
assumptions. One of these is the assumption that it is important to know 
the answer to the question of the exercise. Surely, unless tlin BludGnt is » 
going to be a landscape gardener or fonnlaiii builder, we are interested 
in his abiUty to answer not this particular question but this kind or class 
of question, the basis of the classification being the objective to be 
measured. Let us assume, however, that it is important for the student 
to know the answer to some particular question, for of course there are 
suclr questions in science. Granted this, the second false assumption 4n 
the argument which we are criticizing is that it is important that the 
student know the answer on the date of the examination. For if we are 
interested in the retention of knowledge, in the student's, ability to 
answer the question some months after the examination date, such 
ability will depend crucially on how the student was able to arrive at 
the correct answer in the first place. 

We freely admit Ahat to make our point clear we have chosen groui^s 
^that are extreme in their difTerences. It is nevertheless quite generally 
true that a given test will measure diflercnt abilities if used with dif- 
ferent groups. It is the examiner's responsibility reliably to determine 
what the differences are and consequently what conclusions from test 
results are justified. The speeifications given to the exercise writer must 
include a description of the group or groups that will take the test. 

NecesHary and Sulficipni AbUilies 

We have argued above that, in general, the relative scores of the 
members of a group will depend only on tlie factors with respect to 
which the group is heterogeneous. For example, our group ninnber one 
was homogeneous at the requisite level of mastery of physics, but pre- 
sumably heterogeneous with respect to the ability to read English, If 
it is desired to know the relative standing of the members of a group 
with respect to an ability, the principal ability of the exercise, this 
ability should clearly be a necessary one for doing the exercise, and the 
group should be fairly homogeneous witli respect to all other auxiliary 
iieeessary abilities. The homogeneity of the group relative to auxiliary 
abilities, that an exercise requires, is best attained by writing the 
exercise in such a way that the level of the recjuired auxiliary abilities 
is quite low, l.e,, below that of the great majority of the groirp. If (i 
single grade is'to be assigned on an achievement tost, it may be suflicient 
to keep the required auxiliary abilities below the barely passing level. 
In all casesj^owever, it is not enough Jo assert that tlie students ought 
to possess the auxiUary abiUties; it is necessary to ascertain that the 
great majority of them in fact do, 
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In the exercise under consideration the necessary ribililiesaroErigli.sli. 
terminology, certain knowledge of sound and Hght and difTruclion, and, 
* finally, the ability to relate this knowledge to a particular iitualion. Lei 
us now assume that the latter is tlie ability we want to measurp, and 
that we have a realistic group consisting ot studeiits who have had one 
year of college physics, including the study of difrraction, but who are 
not likely to have discussed the particular test situation in class. If it 
seems likely that a sizable fraction of the group, say one-fourth, does not 
know what dilTraction means,) the term may be explained or omitted 
from iRe exercise. Either of these einendatious wil^niake the group 
pi'operly homogeneous relative to the auxiliary ability of the knowledge 
of terminology but will make the exercise either longer or less clear to 
those who do know tlie term. For sych a group it may be better to 
write an^ altogether diCrerenl exercise. The most effective compromise 
between a reliable measure of an ablUty^and a ^^st of a practical levigtli 
is determined by the group for which tlie test is designed. 

Besides being necessary, the principal ability of the exercise also ought 
to be sufficient for dealing with the exercise. The following simple 
exartiple will clarify the meaning oP necessity and sufTiciency. The 
student is. asked^ "What is the' product of 3 and 7?" The ability to 
multiply is not here necessary because tlie problem can be solved by 
addition. Nor is it suflicientj for ike ternt "product" must be under- 
stood. It should be clear that it is to make the principal ability suflicient 
or effectively sufficient that the auxiliary abilities must be kept at a low 
leveL If the principal ability 4s; necessary but not sufTicient, those who 
can do the exercise have the;abiUt>^ fmd:*tliose who can't may or may 
not have it. If the ability is sufTieient^But nqt necessary, those who can't 
do the exercise don't possess the ability and those who can may or may 
not have it. In achievement testk)^ both the necessity and sufliciency 
conditions must be strived fopv^titioften one has to be satisfied with 
a more modest claim that thejg^iU|y dested for is likely to bd -yery 
helpful in.dec^ng with tlie ex%|c^ej^ 1 

Sliulents' Meiilal Processes and Valid Uy 

There are statistical methods tjMt Ijqjp us hnd out just what ability 
an exercise meaaures. They 'all invlw^eorrclntioiis between the exerelsf? 
and a test whose validity is^nQwn tj) be high. Such tests are not usually 
available, however, Ibf JhB*more btompjex educatioriiil olyectives, aiid tlie 
exercise writer m^i)tlstiU mainly on his surmise of tlic students' , 

mental activitiei ||Hy^ worki% ^Hfeeyereise, One of the guides toward 
identifying t^€ j^^l^^r^d^^^^^K fofniQl ■analysis of the test exercise,. 
If, for example, iri;/Rn^^jS|^^fl5ti. the term "data" seems to be ap- 
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plicable to the information given the student, and if the responies among 
which he is to choose can be legitimately characterized as **interpreta- 
tions*' of the data, it is usually assumM that the exercise raeasures tBe 
ability to interpret data. Studies at Chrcago (1) indicate, however, that ^ 
students vary a great deal in the method of arriving at the right response. 
Soijie of them read the stem— in our case, the data and the question— 
arrive at some interpretation, and look among the prepared responses 
fori a similar one. Others test each responBe in turn agajnst the data and 
reject the inconsistent ones. If the inconsistencies can be estabh'shed by 
inspecting parts of data only, the right response can be chosen without 
understanding even the general trend of the data. Not only in this 
example but generally, the abilities required in going from the question 
to the responses and vice-versa are quite difFcrent. It is in gerieral 
impossible, for example, to use ^objective exercises for a reliable and 
conBistent difTercntiatioii between the old standbys, ^'Ability toanteirpret 
data'' and '^Ability to apply principles/* It is therefore better to^use a 
classification of oyectives that depends on the nature of the situation 
as defined by both the question and the responses and to use **in- 
terprelation ^of data" and ''application of principles" exercis^& only 
to insure variety (2). - j, — 

Ilomogeneily of Responm'^ 
Although there are many ways in which students attack objective 
exercises, in almost all of them the rejection: of wrong responses seems 
to play an important role. If, therefisre, the exercise is to measure a 
specific abiHty, this ability must be the key one not only in recognizing 
the right response but also in rejecting the wrong ones., Tills requires 
a certain honiageneiiy among the responses. Thus, e.g., if the exercise 
is to measure the ability to estimate the cogency of facts as evidence 
for a theory, the responses should all be correctly stated facts and difier 
among tliemselves only in their value as evideiice. If, on the contrary, 
tlie wrong responses contain data that are factually false but that, if 
true, would have been good evidence, the exercise may or may not 
measure straight knowledge, but its rehability of measuring the principal 
ability is reduced. * 

The ''ability-homogeneity'' of responses has other values that may 
tranicend that of insuring the purity of the ability measured. We 
believe that the main advantage of ubjcctiveHest exercises is their 
• iiiUi' ability to define the problem for the student with d precision that ia 
entirely out of n^ach of essay exercises, Tho dennition niust start with 
<S^^y^^ the item, i.e., the part that precedes the respoiiscs; The stem must be 
so suggestive or directive that it guides thd studont^s thinking into proper 
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channels and -even enables hjm to anticipate the general form of the 
responses. Stems like "Wln'ch of the followhig h true?'* niay.produco 
tension and start some speculations in the student*s mind— -pcrhnpH in 
connection with the preceding i(em=that aru merely distracting. The 
yery first response must Airthur define the prohlem or, in many .cases, 
even complete its definition. After reading it, thd belter students should 
be able to formulate a response that resembles riosely in form and sub- 
stance the right rcsponse=^it may of course be the first one, A^Qd/when 
the student, who has the relevant ability, aomv^ npou the right response, 
he should recognize it as such and give only casual attention to the 
following responses. To tin's end in particular and to incr^ise the general 
directiveness of the stem, the stem may well iiichido hints as to the 
nature of the responses* For example, *'AII of the following responses 
are factually correct. Choose the one that contains the most convincing 
(or best established, or most precise) evidence for the theocy.'' Or, 
"None of the responses telow is strictly correct. Choose the oiie tliat 
deviates from the correct response | through being too general (or too 
speciflCi or quantitatively Inaccurate); The problem of communication 
is a thorny one; every part of the exercise must contribute to the stu- 
dent's understanding of what the problem is and what kind of answer, 
as to form, content, precisipn, etc* is expected of him, Tlie argmnent in 
tliis paragrpph sliould make it clear why wc have avoided using the 
term "distractor," a term that applies with such dannn'ng accuracy to 
/many wrong resporises in the existing tests. 

Homogeneity of responses also contributes to economy of eflbrt and 
time. Time allowance for objective items is usnally indecently short. 
Yet it is patently impossible to lest tliu sludcnt's thinking ability with- 
out allowing, him to think. If the student's problem^ is to evaluate the 
cogency of evidence, he should be allowed to coiiceiitrate oii that 
proble^n without being sidetracked into est hnaling lite reliabiHty of 
data or othefMnatters, Nor should he be on the lookonl for verbal traps. 
The **orilys/^ "nevers," and other words m frequently used to dif- 
ferentiate between 4he right aiid wrong responses should be prominently 
displayed. If underliniiJg such words ruins the exercise, chances are it 
was not very good to b(»giri with. 

Homogeneity of responses relative to the ability measured helps 
define the problem, coriccnlrates the slnd(*nrH attention on it, and 
reduces his tensioii. A similarity in ihe forni of the right and the wrong 
responses, i.e,, in their length, presence of qualifying words and phrases, 
tlie degree of 'technicality, and in other essentially superficial respects, 
should help prevent successful guessing based on the auxiliary ability 
of test-sophistication. Jdeally, reHponses shoulci difVer in riollnng but a 
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single quality; in the above e?^mple this quaiity is their cogency m 
evidence for tlie theory. " - =^ 

TheMesi and the Correct Resporm , „ ^ 
In iialural Ecience, as in other disciplines, it m iiearl^ iinpossible to 
give a itrictly correct answer to any but the most trivia! question in 
less than a page, In order to decide how correct the best response should 
he we muit again recall tliat the main function of an exereise is4o divide 
the itudcnts into two groups that differ in a particular abiUty, The 
response marked right should^Mien have appeal to those who liave this 
ability. It is therefore at best i^seless to increase the accuracy of the 
best respouie by adding qualifications whose absence would not be 
noticed by the great majority of the students, say by B+ students and 
those belo>S For an obvious example, the relativistic and quantum 
jnechanical corrections or qualincations are out of place in a test over 
a one-year physics course. It is of course true that the absence of certain 
qualificaliouB niay disturb an exceptionally welMnfonned student. This 
is tu be preferred, however, to changing the e?cercise so as to lower its 
discrimiiiating power for the rest of the class. For, as qualifying phrases 
arc added to inake the best response more nearly correct, two difliculties 
arise. First, a large number of students become puzzled by the presence 
of the over-refined qualifications, aiid second, imiking the formal at^ 
tractiveness 6f wrong responses approximately equal that of the best 
response becomes more diflicult. The optimum correctness of the best 
response nmst of course vary with the caliber of the students. We are 
thus reminded o!ice more that the really elfective test is tailor-made 
to fit a particular population. . - 

In some exercises it is preferable not to have the bust response correct 
even in the modest seiiHc of the preceding parngraph. It is then of course 
usually desirable to warn the students of this fact. By using as the right 
responsfj one that is not correct but is nu?rely the best of those available, 
it. may be possible to Ibrce the student to do imsre proftnmd thinking. 
For example, the best response in the Ibllowing exeicise is A. 
Emrcise: Which of the folluwing is the best definition of Poteniml 
Energy? (None of the definitions is strictly correct) 
^_^Xhe energy which a body possesses because of its position. 
13_The energy whieli a body at rest possesses because of its position, 
a C— The maximum energy wliieh a body can acquire. 

^_Xhe energy which a body possesses before it starts doing work. 
E — ^The energy which enables a body to do work. 

If^the right response were made more correct by including elastic 
p6t|mial energy, its form might easily become so like that in textbooks, 
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! that the exercise could be worked by' rote ihemory^ As it stands, the 
exercise Is likely to discriminate between those wlio understand tho . 
term, potential energy^ and those who don't. It may be remarked iti 
passing that the responses of this exercise lack in formal homogeneity 
because two of them, A and B, are "paired^" i.e., have a close similarity, 
wJiile tiie irdit of the responses are not. Studertts soon learn that paired 
i^egpQnse^ ^raJTiore likely to contain the right one. 
» '.Teachers are almost invariably unhappy about incorrect answers even 
if 'tHeyitfi're shown good statistical evidence that the exercise correlates 
tiignlyi^with the teachers' own choice of gf)od students. Their arguments 
^gainit! incorrect best responses vary from good to bad, the bad c^ies 
being raore frequent. An exanljde of a good argument' is tliat since* we 
have rip^ accurate knowledge of the mental processes involved in the 
criterito situation, i.e., in resolving a genuine -aOientific problem, or 
thosd psed in the test situation, we should hold firmly to the few simi- 
larities between the two situations that are under our control. Thus in 
a criterion situatibn of almost any sort it' is a/statemenl of the correct 
definition or law that is useful; tlierefurtf feiiriirarty correct statements 
shouldfbe used in test situations. There is a gpod deal of truth in this 
analysist although it should^be Jiioted'that the choice of^i-he Bett ap- 
proximation concernSi,thtvs!sietitist and the citizen more oftdif ;lhan the 
choice between the corrfegt and tlic wrong, A less good argument, based 
on the uncontestable triiilljf that tests are valuable tools qfi instruction, 
claims that therefore they sllijuld contain nothing butithe^iruth. What 
the student is likely to learn from a test, however, is rtqf 6n isolated 
fact cfr generalization, for the te^t is encompasajd in a few pages and a ^ 
few hours as uompared to the hundreds of pages of the text studied over 
a period of inonths. The tnoHt effective role of the test as a teaching 
instrunient is rather to let. the sOident know in rm emphatic manner 
what the objectives of the^onr.He are. Another c|nestionable argument 
runs us follows: "T told niy stndtnits that *the energy which a body 
possesses because of its position* is not a coiieet definition of potential 
energy. It is not fair to ask theni to act r-pt it.*' Our reply to this teacher 
would be that the only physicsMtest that is fair to the students 
is tlie one that reliably identifies those of them who understand physics. 
The question is rather whether the exercise in question is fair or kind 
to the teacher. On this point we would say fiist, that fallibility of 
teachers and texts is not a bad thing for students to learn, and second, 
that the teacher should have given his class a deeper eriticism of the 
quoted definition thaii 'merely calling it wrong. Whatever the worth of 
the arguments against exercises with incorrect best responses, it should 
be clear that writing such exercises does require a steady hafid. 
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T}^ Wr0Rg Responses " 1. , ., 

The preyalent criterion for.; a wtong response seems to be that it 
should be wrong and yet plaUMble. As we have indicated before, jhjs 
requirement, although necessary, is not enough. If the student is asked - 
,tD choose among conclusions from some data, the wrong respbnses • 
should be wronf only in their relation to llio data. Of course even ifv , 
the wrong, responses are homogericpus in tliis respect, tlicre are still , .; 
degrees and kinds of wrongness that are possible. The degree of wrong- " 
■ ness determines the difficulty of the Hmu and sets- the main discrimina. 
tion Une, e.g., between A and B or bolweon D HndjP sludonts; ThoJtmd - 
of wrongness determines the auxiliary abih'ties tha^ aw; helpful in finding 
the right response. Tims, e.g., if tlie wi ongnoss in the preceding example 
is that of goini beyond data, students who have' such a weakness will 
be more attracted to the wrong responses than, say, students who have 
the weakness of being over-coutions. Bcsides'tlitse two, there are many 
other weaknesses that make wrong responses and Bometimes the right 
response, unequally attractive to students who possess the principal 
abiUty in an equal degree. Two confiinon ones are di'sliust of theory and « 
' over-rQlianqo .on quantitative data. A piny oil various students' weak- 
- aiesse8;tti.make wi-bng responses attractive to Mieri> ip pftcn justified and^ 
• itt ty;t,^ttnMldom;t)e avoided. It is iiet:ossaiy, however, ^hat ilie -preyn-; 
JentWeHkiie^se^b^sWiipIod toW- Such sniiipling is controlled prinwrily 
by the nature pf iiiLwi'ongrt^s^iJnses and to a smaller degrde of the right 
resironse if it dovifttes, lis it usuajly must, fronvtlw strictly c6rw<5t one; 
/ Although it is difficult, and unnccessnry, to hav\3 a single exercise 
discriminate at several levels of SbiHly, it is quite easy and usefuUo ; 
make it discriminate at two levels by uiukiitg one or two of the wrong' 
responses so wrong that only failing students isliould be attracted by it. 
Let ua call such a response an F-rcsponse.: and tlie riglit resppnse, 
R-responscLet the miVnber of F-rosponses a itudent chooses be called 
his F-seore, and similarly for R.-It will almost always bo found that the 
R-minus-F scores are more reliable than the R-scori^,; F-scores 'may 
discriminate more reliably between F-students and i the -rest than any 
otlier scores. It sliould be noted tliat for studeiits ^loj^ake a try at 
every exercise, the reliability of the usual right'-iniiiusiwrong scores is 
liontical with that of the R-scores. F-scorvSihrMlsp useful for establish- , 
Ihg isbtolute standards for passing perforriiai^^^^^ 

^- ' Spontaneity , 
If the strongest argument for objective tests is that they make it 
possible to make it cloar to the studtot what expected of him, the 
slrongeit argument for the essay te^t^we are not here considering the 

0,5 ■ . ... 



I 



ERIC 



' * ^ 
* . ^ . -I ' 

ability to express;pheielf— is that it calls for a spontaiieously produced 
answer. Sine© all .critefipiv situations cpill for ^greater spontaneity than 
is required in the^s^al y^jective test, the latter, if not supplementeid 
by an epsay teitV i^ust be 'hiodified to decreaste such discrepancy in 
some pf^jthe ^xerffisfes. 'Art- objective axercisd th\t comes closest to this 
r(^uirameiVtV is one iii student reads the statement of the 

i^lPdblerurdeydes oh'th^ answer, and then searclies for it among 
; the prepared responses. Such a procedure can never be enforced but it 
can be OTade inore. profitable than any other. To this end the steui must 
be very dijfeGtive and the responses nqii-diri^tiveorev^n non-eyocativei 
i.e.* non-suggesjive. . , 

The following excircise has non=evocatlve responses. The student is 
asked to solve tw.o iimultaneous equations: 3x + 2y — 17, and x--y = 9. 
If the responses were pairs of values of x and y, some students would 
find It easier not to solve the equations but rather to substitute the pairs 
of values into the equatiqns. If, on the othc^r hand, the correct response 
is given as x+y = 5j and'tlie wrong ones sifnilarly, a solution of two 
sipiultaneous equations ii tlie only way to tlie riglil answer, and solving 
the given Ivvb is by far the quickest way, 

, ; J?unierical science problems Icmd Ihemselves welMo the technique of 
ncin^evocative responses. For exaniple, each response may list just the 
second digit of a numerical answer. 'Or, tlie studejit rnay b^vasked to 
compute two quantities, or two formulas, and asked to choose anriong 
responses .that are ratios of the. tw6. In qualitative exercises^ a similar 
result tti ay be achieved by making the analysts of each response, withoiit 
fir^ Sblvujg' tb6 problem df^^the exercise, as innch of a chore as solving 
the pfoblehii^ T^&^follpwing' exercise, in which, to save spaeCi we show 
only twd f ^f^onses^ ^ i^^^ 

Baercfee.'^ .This Ipxercise invorvcs. two steps. First, decide wjiat law of 
nature is most directly useful in explaining ihQ following fact: A brick 
can be pulled' along ji fairly sinooth surfactr^by'. means of a string; tlic 
string would break, iiowever, if jerked HharpiyiBecond, choose that one 
of the fdllowirtg ^phenoinena for w liich this h\w t)f nature provides an 
explanation. ^ 

% — A glass tube drOTpedTroin a lieight.of ;10 feqt breaks if it falls on a 
concrete sidewalk but' will' not hreak ff it falls ort a soft ground. 
B— It is impossiblk td lift onq|elf by pulling ftp on one's own hair, 

Most students willVfiifid that the na)st enic.ierit way to deal >vith tha 
above exercise is to* follp^^'tlu;} directidns'-ivhieh suggest giving a spon- 
taneous Answer to the first t|uestio!h Dirfections to the stuUqtitsjniist of 
course alvvays be scrupulouMy honest in the sense of iridicatifig the. 
easiest path to, the right response. We note tliat tlie second part of the 
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/ Btem m the al^ve.ax^clse is not at all directive: This is the. usual price 

int^ ohjectiyc excrd^eB. 
•>ff Ynotel^a^^ combine Uu^ more advan^ 

tftgfious ftBpf^^^^^^^ ftsaftf arg^ objective exercises is dnRcrii M in 



anbther article (4). 

^ ' / . Snmmary , ^ 

An eflfective test must be tailor-made for particular population. . 
" ^ The degree and range of the students* abilities MeCermine: the opti- 
inum corwtneM of the best response; the degree |nd kind Af .wrongs 
\ of the. wrq^ re^onses; the leyglof the auxiliary abilities necessary 

for working ihYexiWse. This lev6l should bo below that of the great 
maiority of th0 students' ^ 
/ . "ftie defini'ttbn- of the ability meaiured by au q>^^cise should hold 
true for all the prevalent methdds of soUilion used by studpts; e.g., the 
method of eUmihatihg wrong resffonsus. " . J/ ^ ^ ' 

' "The principal ability of the exm^nise should be both ,rycnsHary 'and 
sunicient lbr working the exercise.^ ^ Vj 

^ .] Test exercises meflsuring a particular ability Bhould exhibiFoi grccK ' 
variety as isconsormnt with the definition of the ability. They can vaj-y 
in content, form, type of onalysiH required, dinieulty, and auxiliary 

abilities, . ■ ■ ; . 

The situation on which an exercise is'bnsed shiulU not bo similar to 

the textbook ones. ^ ^ ^ 

Abilities auxiliary to a group of iteftTS should bo samplecl ftiirly by these. 
"I'he gtem of an exercise should'^ tfohtain .s> of the 

problonl and even suggest the desired ki?id of solution. ^ 

Responses should be hon^genroul i^lative^to the prinnipnl nbllity 

. and inr their form- ^ ^ 

^ The technique of, using inco^ect besl^pohses is us 

It is easy ana^desirable to include among wrong TCsponBes some thnl 
* ara^ytiiflctive to faihng ^ 

Iti^ boa^ible to make an obdoctive exerci^,evokc in the ^tiulc^its . 
nearly spontaneous answers by rnnking thin process the most eeommiiral^^ 
one for dealing' with the exercise 

^^^m 1 "Fofmiilatlon of Oyeotives of Teachins in the PhyHical bcimiccs, by Lk<i 

^■■r iNEDBLSKY. Amencan Jourml of Physics, 17. p. M3, Sept^mbcf 191f9. 

^ ■ 'I Bee the author'B arUcles in Edacaiional and Psychological Measiiremcniy AbsQlut4* 

to ■ ' Grading Sl^ndardtf for Oyective Tests," Sprmg. 1954, p, 3. and /Lbibty* to 

■ A.void Gross JEtror as a Measure #Achievenian^ Autwin, 19q4. p. 459. 

1 '^Evaluation of EssaVB by ObjicTive ^ests/ by Leo -SmmiMKy. -./nnmnl nj 
f^fft™/ £rfiiert/mn. Vn,.p, 209, ApriM ^ 
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' In ^scussing the aft of exercise writing iii the sooiai scienceSi it seems 
wise to limit tihe discusiion to exercises useful in Bvaluating certain im- 
jportant objeptiv^s of a college level general cour|i in social scieiice, 
Ttfafay of the suggeitionstnade also have appUcatioh to more specialized 
^S^SSm^m^^^ Wb^^ fieid on both the high school and college 

lefVels* Baforis conddering the Writing of exerciies, it is desirable to 
characterize briefly /what constitute in my* judgment, dean'oble instruc'^ 
^ tiohjpl pbjegliyes, naethoda of instruction^ and subject-mattor contDnt of 
Va sdbial seieiice general cour^CH ' # . ^ 

»^ I . . \^i'mtmclional Objectiiks in Social Science . 

As in' other ^gubj^cts, the instructional objectiTes %t such a course 
may berilassifled under the headings of (1) knowledge^ (2) intellectual 
sklllSj aAd (3) ideals,, attitudes, interests^ and appreciations. Under the 

_ Jfrstjiepdm^^ or understaiiding of spjpciflc facti, 

terminology, and principles. We may also include knowledge of the 
niethods of inquiry in social science and how social science can contribulo 
to th^ making of choices betwden vahies^ without determining, as sciehce, 
which valiieS to choose. . 

Under intellectual skills, we may include the skills required in making 
discriminationB and comparisons and in organizing knowledge in ways 
wliich contribute to understanding of relationships. We may hiclud^^ 
the sKIlls requ£rted in reading discussions of social probleuis or issues 
with comprehension ^and with sensitivity to ttie logic, or lack of logic, 
of w^at is read,rpresuming that these discussions are of varying points 
of view, or are based on evidence of varying relevancy and dependa- 

j biJity. Under intellectual skills we may indude the ability to analyze a 
social prpb^m, to recognize kssuniptions and to propose hypotheses, to 
obtain releyant data from 'appropriate sour^s and arrive at war- 
ranted conclusions-. Especially important is the actjuisition of the 

vability to predict and to compare the consequences of difTerent couraes 
of action. Thiri objective is fundamental to the critical evaluatioti, of 
social poflcies espontlal to eCTective citizenship* . 
^ There are, of courae, no sharp boundaries between the levels of 
objectives juat-^inentioned. Tins is also true of the classification pf 
objectives in tftg Taxonomy of Educational Objectives by Bloom and 
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otiiett<(2)* Furthennore, an ekercise may evaluate the attainment of 

*i dtferent ipegiflc 'objectivei' givto different learning experiences. 

' " We B^U not here be concerned with the formulatinK of essay questions 
nQis4H*h**'" «v«*»""t inn'of ideab. attitudes, interestsffnd appreoifltibns. 

f In the bibliogr^hy of this paper the speaker has Listed the provocative 
aVticle if V. M. Siroa ^titled "The Essay Question is" a Projective " 
Tftcltoique" and John Stahiaker'B scholarly chapter on essay examina- 

' tioni in Wndqulst, el al., Educalioml Measurement. (IB) A compre- 
hen^e prograifl if njemsurement in a social science general course 
%ould include lome amount of essay testing, the writing of papers on 
socW probes, and the evaluation of change in-,B0clal attitudes and 
^eUefs. Levi's ^eral ^dumlumn Ihe Social Sludies (8) is one source 
of infoBira^ten with respect to such evaluation. . 

! ' d&nl and Methods Social Science I m^mlion 

Vhere teaching is reatricted to the imparting of the content of a 
te^t irf fai^ess to the . students, evaluation sliould be restricted to 

- measurement of t^information they liave tlms obtained. On the other 
hand instruction wht6h justiries the uku of exorcises mcasurmg intel- 

. lectual skills sht^ld prdvide for the devolopment of sucli skills. In addi- 
tion to the explicit recognition of tlicso skills as objectives, inslruction 
should give student? numerous opportunities to acquire and practice 
• them In dap discussion aiid in written ftssigrunents, Ihought-provoking 
problems should frequeiitjy bo presented., Thr instructor sliould ^ 
constructively aitioal of the tliinking done by students wlietlier m 
recitation or in written work. WJiile mucli timo must be devolod to the 
impdrtlng of knowledge/by an inslructor and to its aequisition by 
Btudents. instruction can inclado dovelopriic.it of understanding of the 
nature of assumptions and liypothescs, and of tlie eJeineiita of logicaj 

. ■ reasoning. Studehts should be expected to use torrtis with precision arvd 
to support their answers witji evidence. Problems and issues Bhould-be 
analyzed in class'so that studonts may be trained in identifying problems 

' or i|sues,"in recognizing assumptions, in determining what kinds of data 
are heedfed t^ suppoft or disprove hypotlicacs, and wliat methods are 
- useful iS' collecting and in interpreting the data. From tun^ to time 
instruction should include the comparing and contrasting of facts, 
ideas, principles, *nd generalizations enrlior_,learned.with those being 
■ learned so that the student is able to orgahize his knowledge in ways 
which will proye helpful in solving^now problems? 



*The numbora in parenthw.s rcfrr U. ll.f nurnb. n.d ilmnsln lh<- r.-foit-nocs at thi> 
(■lid of the papfr, * ' ''^S^' , 
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WhUe' there are numerous iources of information cpnceriiing the 
methodology of ioeial ^lenoe instruction, refarence will be made only 
to two r^ent and chaUenging books^ — Theory and Practice of the Social 
;S^ad£ef by Earl John§op'(6) Bud Teaching High School Sociial Studies 
by Maurice 'huAt and Lawrence Metcalf* (5) The latter ahould be of 
equfd interest to teachera of the social studies on the college leveL It 
is unique in its applications of field psychology to leafning in social 
science. It strongly advocates the use of content relevant to areas of . 
conflict and of contradictory beliefs both within persons and between 
peisons such as social class; race and minority group relations; sex, 
opurtahipt and marriage; and reUgion morality. Whetbepor not these 
particular, areas are included, Urn content of the social science general 
course should include areas in which there are iraportant cdntroversial 
problems and issues. Contemporary Social Issues by Leej Burkhart* and 
Shaw, (7) a^d Basic Issms of American Bemocracy by Bishop and 
Handel (1) are ^anaples of sources in which students may read selections 
presenting opposing points of view on numerous problems and iaiues. 
Society and Man hy Weinberg and Shabiit (11) is unique in that its 
authors have shortened and rewritterij in language understandable to 
students* but accepteible to the original authors, research studies and 
basic writings of noted authorities in the social science field. 

Unless the atudents have had previous instruction relevant to funda- 
mental concepts and principles of sociology, economics, and gbvernment* 
it is possibly unwise to base a general course on such a "problems" text 
alone. There may be, however, concurrent use of a systematic text and 
a problenii text. While ihstruction and class discussion should be con- 
cerned with problems and issues, it is more effective to deal intensively 
with a relatively few problems or issues in relation to more systematio 
content and in relation to social science methods of inquiry, than to 
deal with numerous problems or issues superficially. With all of the 
preceding as introduction^ let us turn to the art of exercise writing in 
the social sciences. . 

Tlw Bole of theTfacher in Exercise Writing 

While it would seem logical to insist that exercise writing begin with 
a carefully formulated and detailed list of specific objectives and a 
detailed analysis of the subject matter conteul to be covered, it is my 
experience that such a fgrmal approach is seldom effective with a group 
of teachers. It should be emphasized, however, that a less formal ap- 
proa^ with tlie underlying goal of gradual development toward ade- 
quate definition of objectives and representative sampling of content 
can leadp after a number of semesters, to the production of exercises , 
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cbmpatlble with a wide rang© of objectives. There will be an increasing 
empharii on inteUaetual ^Us 'or critical thinking in the social science 
field both ttt instruction and in evaluation of the goals of instruction. 

MtCT numerous ^^cisas have been written by the various teachers 
of a group, its me?nters should criticaUy evaluate each other's wdrkr^ 
When Afferent Instructora key exercises independently, discussion of 
disagreements with respect to correct answers can stimulate improve- 
> r ; ment of the^content and phraseology of the exercises. Group evaluation 
' of exercises may result in the elimination of exercises relevant to trivial 
content or to content uniquely taught by one of the teac|iers. Group 
evaluation may also r^ult in the organization of an examination repre- 
sentative of the content of the course and of the objectives it is desired to 
evaluate. After the examination has been given and subjected to item 
analysis* the item difDculties, expressed as per cents of correct response, 
make it possible for each teacher to evaluate acWevemeut in terms of 
specific objectives. The item difficulties and the Item-test correlations 
; are also of great help to teachers in revising series of exercises for future 
use. It is especially true of social science exercises that knowledge of 
the per cents of response to eacli answer, both correct auH incor- 
rect, contributes significantly to the imprbvement of faulty exercises, 

Tim Wriling of Mnliiple-piqice Exerc 

As in other fields the writer of nmltiple-choido exercises in social 
science should observe a number of precautions.^ie introductory part 
of each exercise, the ''item, stem" should present the problem of the 
exercise. This promotes both clearer understanding of the problem of 
the eseercise and the use of briefer answers, which often may be no more 
tlian single^words or phrases. Such exercises are more economical of 
testing time and'since more exercises can be used the total test may be 
s mose valid and reliable. Unless the exercise begins with a question, each 
\ answer should complete the item stem grammatically. One of the 
\ answers should be definitely correct and the other answers should be 
\ plausible although incorrect. (In exercises of the best-answer type^^ne 
_y of the answers should definitely be^the best) Frequently incorrect 
answers* or ''distractors" can be true in themselves, but not relevant 
to the problem set by the item stem. Often an exercise writer can produce 
good distractors by anticipating the kinds of answers students are likely 
t^ select when applying the wrong information or the wrong kind of 
thinking to the problem. Precision in the use of EngUsh and the ability 
, to express complex ideas in brief and sirAple phraseology are desirable 
attributes of an exercip writer. Some of the suggestions just made may 
be illustrated by quoting two multiple-cl\pice exercises. 
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1. Authoriti^ on maiTiage deplore the American "ideaUstic belief in 
fomantio love*" A young couple characterized by this belief will 
be more Iftaly to have a succeasful and happy mrfrriage if also 
charactimed by , ^ 

A^ a belief in the double sta iidard of morality r 
B. a wHUngnete to take rides. 
C* Emotional maturity and adaptability* 
D* a beUef in the contractual concept' of marriage. 
- E* physical attractiveneis. 

2. AJarge increase in total spending by consumers^ businesSj industry, 
and^fovernment will probably not cause inflation, if 

A. fbvefnment bonds are sold only to individuals. 

B. interest rates on loans are reduced i f 

C, production of goods increases in proportion to spending. 

D, less money is saved by individuals. ' 

E* money is placed in circulation as needed for spending. 

~ Clmsificaiion or Key-LiM Exorcism 

While many and sometimes all of the exercises in a bqcislI science 
examination may be of the multiple-choice type and * 'self-contained" 
in the sense that a given exercise is .not a member of a series of related 
exercises/it is frequently eff^ to use series of items of the classifica- 
tion or key-list types. In my judgment, items of tli^e types can be 
useful in evaluating understanding of relaMonships and ability to make 
comparisons and discriminations. The expectation of such series of 
items* ranging more widely over content than self-contained multiple- ^ 
choice exercises, can motivate students to organize their knowledge, an. 
advantage sometimes claimed for essay testing by persons critical oj' 
objective testsv 

Suppose, for example, that in a social science course, instruction Iuuh 
been concerned with the characteristics of liberal democracy, com- 
munism, and fascism, and that the instruction has included sonic 
discussion involving the contrasting and comparing of these ideologies. 
Then a series of exercises such as the following is appropriate. 

. After each item number on the answer sheet, blacken one lettered 
space to desigMte that the item is characteristic of the theory of 
/ A Uheral dem^racy. 
B Communism. 
. . C Fascism, 

D both Communism and Fascism. 

E both libefal democracy and Communism. 
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r 1. There should be reject for individual personality in both the ends 

'Md means of foverament* 
2 JVediom and equaUty are ineaningful only in a classless society. 
J . 3. A tempoyary diqtatoi^p may precede the estabUshment of a 

"statel^" sqciety, 
4» The ultimate goal is freedom and equaUty for all in a democratic 

society. 

RTOill of the comparisons made during instruction may enable many 
students to clarify such items correctly. Something more than mere 
recaU is iieMed, however, if the items are not in the same words used by 
the insteuctor and if ^e studMt has to^select the ideas that are relevant 
from ft wide range of information. In writing such'exercises a number of 
precautions should be observed. The categories should be related, but 
mutually exclusive. If, for exanaple, the categorlei pertaining to liberal 
democracyi communism, and fascism had inclnded socialism it would be 
difficult to classify such an item as Advocates collective ownership of 
. / tbe means of productipn/' Careful wording of the categories is extremely 
. V >;}V,Mportant.' Thrf^ word "theory" was included in the directions since it 
; a ■ Wj^ 'eviden^jO tt^ this series that the students^ could become 

/, ^ ' qohf^fed' if.|hk;^oiliht of practices rather than theories. The items 
'^!v ' H should not,Wfong, involved, and qualifled complex sentences. 
- - In thbiAl some controyersial problem or i^ue in 

the '6oG4af ,.fieia,^^rt^^ to become alert to arguments that a 

' ' prot^i^rti^^ne dd^ of tte issue will use to support^ h^s side and the 
arguments The will ft^VaMce a^in8t#6. bther side. Consider in this con- 
nection the following 'key-i^:daj^a^s^ a few of the items to be 
classified in accordance with them. '' / _ - 

Imagine two persons debating the relative merits and limitations of 
presidential and parliamentary governments. After each item number 
on the answer sheet, blacken one lettered space to designate that the 
item is an argument advanced 

A in support of presidential government* 

B in opposition to presidential government* 

C in support of parliamentary government. v 

D in opposition to parliamentary government. 

1, Such a government is especially designed to prevent a dangerous 
concentratipn of power. 

2. Such a government should respoiid more quickly to the e^^ressed 
wiU of the people, m ^ 

3* E3qj^ence shows that a governr^nt which permits the executive 
and the legislative to become rivals for power retards govern- 
mented action. 
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4, Such a government is esp^iaU); .me^ several 
pqUtical parties. - 

In evalualing intellectual skillsij it te.^ff^livjs; sed^s of multiple- 
^dce ^ercises or key-Ust iten^; relent iti^^M quoted 
within the examination itself, or toigii^ j^igtt^^^ reading 
prior to the lamination. Selectibns j^^ of view 
or advocating different courses useful On 
oocasion, exercises fbUowing j^; :bH0f;';Wlec^ for student 
analyA of its content in relatiort thtt^^ selections * 
earlier assigned. In a recent ^kanunati^^^^^^^ American 
Community Behavior by Jessie J^rj^tt^ ftitr^utes^^ grpup con- 
flict to competition for jobi.ra!i4;t\f0ire0 cbh&i^^ including 
housing. Three of the multiprle/^ii^ip© fotldwltig tliis paragrapli 
expect the student to compare ili ttidjiis 3^ of view on 
minority group relations expi^^^i^^B^ert T. V, Smith, 
and Gunnar Myrdal in selepti^>nfc g^j;er;'s^^^ of the exercises 
follows: * r '' ' ^^v-^ ^^^ ^^'^^^^^ * ' " 

I. In his American i^eVeMt^ff/jMyrtffjf dip the above writer 

■ ' -in that ftlyrdal ^ rj,:^ /" ' ' ' ^ 

A. opposes racial diicrimtnatioi^^^^^ , ' ^! 
V B. explains discrimifia tidn" -iri ^tP>^' of * inconsistent social Valbei?. 

■ C ignores the i*plp'6f egoiloniic facto riS in causing racial prejudice. ^ ' j 
^ '''^^ ftvdBs *av certain' degree '^ m^ 

Each answer in the fpUd^iii^^^Tcerci^^^^ from l.ir'^ 

Jefferson's^flr^t irtaugUraI"acl^reis;p^os^^ above the expt^cispSr . . , I- 

U. Ifi view of Han1il|c!iin-s>BeU to our ' govetnmBryt^ it . ■ , ^ 

, - i? likely that he 'wpuTd coMider " i V - 

V A* ^'fiqual and ^^act jus^ ^ " ^ 

B. v^*The suppbrt if ille state |ov in all their rights. V- :-= 
J C , ■ ' *Th e supremacy ^ M-i^lm civil oyer lli e m i I i tary a u t lior! ty , ^ ^ , ; ^ > . 

D. 'The honeft payrperit ^ >5*: . . 

*Whqn two or threfe brier§^lQctloris\ad^ contrasting pbints of 

view* or differing cdutsei of; adtt^ within an examinatiiqifc 

they naay be followed ■ by iHliliip^ or by key-list itemp."^^*; 

Fdf- example, one/seieclido:Tnay^^^ from Edmund Burke's • : 

■*Addreis to the EJ^dtqri of,S^ri^tor' .while the other may be a paragraph 
»^by: Speaker Rayburti QU Uielnipotlance to a legislator of respecting the « 
, wiihes of his conitittjents. A ^erlei of itoms tnay fol k)w such directions as : 
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For mch ^ At toUowing items, blacken one lettered space if the itein 

is ont irtUi which ^ 
A Burke would agree. * 



0 botii would agree. 
D neithOT would agree. 

Certain thinp need to be considere^ n selec tin g materialfor quotation 
wit^n a sodal science examination. In addition to relevance to instruc- 
tion and elements of novelty, the quoted material should not be of 
reasonable length. . It should say much in Uttle space. In some cases it 
ii necesaary to adapt rather than to quote literally. Often it is desirable 
to substitute words or phrasei more readily understood by studentf 
On one occaiion/**the. architectural droyiiig of a benevolent welfare 
state*' was changed to "the idea of a welfare state/* Fr^uently, modi- ^ 
fication of the quotation may be guided by the exercise/ written. Senj 
^^s which do not contribute to the iolutlori of any exercises may ht 
*bmittfed if this does not impair the major thought of the, selection. 
Sometimes an exercise can be keyed more re&dily if a word or phrase is 
modified. When a quotation has been changed as described above the 
citation of Hf;aqu^^6ffl^;B^^ 

Some of S'Sxerak^^ipIr^ to quoted selecUwB may evaluate 
*^ackground*^ knowledfee of facts, terminology, principles, or .con- 
ditions relevant to the quoted material but not defined or explmned 
therein. While certain of the exerciBes may be Written to evaluate KhowI- 
edge. other exercises may be wTitten to evaluate the acquisition of the 
intellectual skills characteristic of critical thinking. For example, nil 
qxercise call for the identification of a central issue or problem i 

i^Th^^as0^robloH goverrimenl involvp^ Atv situation 

described'^bove is ^ I : , \;: • 

A. , free enteripfise Mp.^iocialism. 

B. democracy vs. dictatorship,' 

C. individual freedom vs. the welfare of society. 

D. government by an elite vs. government by the nms^es. ^ 

Another exercke may have to do with the identification offan assump- 
tion or an hypothesis. For example: 

1. The above paragraph deicribes how interviewers collect data to 
* be used In prediotirig which party will win an elecjion, The basi^ 
assumption made is that . . , ^ 

A; the persons interviewed have sound rea^oiis for voting as they 

pian to do. 
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tte pmmm intemewed are well iflfonn^ with reject to the 
merits of ^e candidates. 
, C* aU aga *poups in the populatloQ are represents. 
Pj^ tte fiSlPM iPtervigwjrf^are repr^ntative of all the voter fl on 
eleotion day. , 

Ebcamples are given below of exe|cise3 oaUing for the identification 
in a quoted sel^tion of bias, prejudice, and propaganda devices; of in- 
GOnuBtenci^ in an argument; 'and of ^limitations in th^ data presented,^ 
or in the technique us^ in collecting them. 

1. In^de^ribing Mfv Brown'ii^^^m one ban justiflably say that 
A /lie was consistently factuaL 

B. he was logical and precise. ^> ' 

C. he u§@d, pn^|ganda devices. - ^ v^; 

D. he dWufeed both sides of the issue.i ^ ^ 

2. Although^ Mr, Brown is for "free entliprise,*' he is inconsistent 
in a^ing for ' 

A. an elimination of the 'excess profits tax. 

B. reduction of government expenditures. 

C| eUmi nation of subsidies for farmerp* V ■ 

D/ government insurance of home loaiis Aade^^^ ag^cies. 

3. The data reported in the table shdw^ that ^/grealti^^p^oppj^ 
childpen from brdK^i^ homes become delinquent than ;pther jdh^fliir^n . 

, Thi importance thia factor could be better evaluated if ^e fcnew 

A. the comparative imtelUgence of deUnquent and honnlfliiiquent 
cblldr^. Vfi - V . 

B. the ^onomic status of the homes of both types of children: . 

» C. the educational oppbi^tunitieB of both types of children. v v f 
D. all of the abow* ' ^ v - 

4. The author of the experiment summarized above concludes that 
children learn more efficiently when motivated by praise rather - i;'>^5^ 
than reproof. Wo would-be more confident of 4he resulta of thi&^^ 
v " experiment had the ^xperiinentiBr ; s ' ^ 

/A. had the pupils alternately praised and rcpiw 

tion. * '* ' ' ■ - ■ \ '. 

B. used a control group of pupils of equivaleht Initial ability sub- 
jected to reproof. 

C. used an equivalent control group neither praised nor reproved. 

D. tested a diflFerent hypothesis. 

Certain exercises may require discrimination between ' expressions of 
f|ct, opinion, or value judgnfenti: 
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h InsteS of itftting a fact, the ^tence in the paragraph which 
pr^i^ a belief or pinion be^ns ^ j ^ ^ V 

A. The price of agricultural products . . ^ " 

B. Hie goy^w aent sho uld ■ / . \ 
C The swplus of ^h^fi*" 

D. Last yeaTj exports of wheat . , . ^ . 
Other ^ercis^^ may caU for identification of iiiferehces supported or 
qontraActa by date in a teble. a graph, or iunainarized m a paragrftpb^ 

1 . The data given m the graph could be used to argue tl^t ^ 

A, indi^teial profits are too high, ^ * 

B. priee controls should have been retained^ lonP after the close 
ofWoridWarIL ] 

C. the supply of consumer goods has decreased in recent years, 
behcep prices ar& high, ■ ' ^ 

D, a deprfssion is inevitable, , 

2. Judging f^m the flares reported In the article, it is correct to 
conclude that 

A. consumers are buying more gopda, but at lower pr^s. 

B. consumers are Buying le^goods, ;^ jUv 

C. manufactures intend to mcreaefe their sales el^tlsv^^ 
^ D/one-fburth^tpmanufacturer 

customers^ati plentiful ' ;^ ' ':. >^ 

An exercise may evkluate the ability to recugnike the iH^ed for addi^ 
, tional evidence of of the effect of new evidende ii? acqepMng or r^ectmg 
^ome conclusion stated in the quotation: 

I One could more justifiably accept the conclusion that slum children 
_ ; are lefe intelligent than children from more privileged environments 
* if we knew that the investigator had ' 
^ A. tested larger samples of both types of children. 
V B. used a test tliat is fair for both types of childten, 
■ C. done both of the above. 

p. done neither of Jthe above. ^ 

J 2. Wo could most justifiably accept the author^s conclusion that J^e 
Taft-Htotley Act has served to "enslave" workers if we knew thai 
A. la^r leaders in general oppose this Act. 
B i^nions of plant maikgers were obtained, 
C the status^ of workers bad actually declined since its adoption. 
} t) . a survey was made of the opiftions of a representative sample 
of workers. ' ; , ^ 
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Frequently, an «cercte shouM caU for the Identificatidt! of tlie^ 
qimcM^ effecti of pven purses of action. For example; 

L If salM tax advocated by the^authQr of the above quotatioh 
|were to ba/^^ted, wlu^ fell Qwin g would be lUMy k 

happen? 

A. I^s money would-be collected by njeans of the income 
B; People Mth la^e i^bomes would suffer more tlian pe| 
^idl income. 

. C. People with small inqdtnes woul^ suffer more than pepj 
large ineomes. 




,v^; D. lbs burden of thb tax would 1^ proportionWe jto tacitoW^ 

Conclusion : = . A ' 

A ^variety of types of objective pxercjaes have been discurted Imd 
iUustrat^ and numerourf iUggestibns have been made concernuig the 
art of writing them. An Effort also has/been made to emphasize relation- 
ships between sodal icitnce instruction jand evaluation. The questidh 
may be raiapd as to how Wp know that; such exeTciies measure more than 
r^aU ihformation. It is evWent from Jitudy of the exercises: that knowl- 

f edge^ia an important faqtqr irf lh^|^:8blutto^^ It seems tq me 'that it is 
al^ l^titoflrte to infer' that manyHof the intellectual skills functioning 

:: to-eritieai thiiiklnf are ab^ evaluated, aisunnng that social science 

: ih^ruction has ^een of the character described. 
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Arthur E^ Tt^er, jrWayne ^i^^ ; . 

CHAIRMAN WRlGHTSTONEr #A have a few«iinutes for ques- 
tions, and ftlthbugh I am basioally a friendly person, I have been ad^aied 
that I Aall have to prompt you if you do not speak loudly enough to 
that you can be heard by the itenotypist. I shall carry fcffward thu 
aaaipiBiAt. and prompt you if you do ndt speak loudly enough. 
■ DR KOSTICK-thls isfor Dt.Ncdelsky. , ; 

You suggested that an erroneous choice that is too sweeping is no«' 
good in so far as it penalizes a particular group of students. We might 
MY thB group that is penalized subscribes to Bweeping statements while 
the group that is too conservative would have an advantage on. such 
«fl item. Now another example would be a choice that would not be 
broad enough or sweeping, enough. This type of item would penalize 
tfie group ttat thinks too broadly. 

*My question is. how do you feel about usliig a five choice item u. 
which there is ope best choice; the four 6ther choices penalize four groups 
^that anrbiaBed in diflferent directions? i • 

DR NEDELSKY : I definitely agree that would be a proper solution. 
You can do it if you are ingenious enough to have it in one item. But 
^it U perfectly all right to„penaUzfe one group in one item, and anqttier 

one in anotter. . , ■ _f 

DR CARSTATER: T was -struck by the aptness of the story ol ur. 
Nedelky's treatment of the archbishop's question. On being pierced,^ 
each of the speakers gave forth two substances. These iLlustrate the \ 
Inter^ipUnary approach that has prevailed in these 
Dieitosh was to talk on the humanities, and a large part of bis staie- 
me^W to do with social sciences. I couldn't help observing that 
Nedeldif was getting into semantics, which seems to me to be a part 
6f the l^uraanities. And ^large part of Engelhart s presentatton ha^ to 
do with the scientific asp^ect of the social studies. I thmk this is quite 
desirable, that there should be this interplay. ,u 
One other thing about the aptness of the illustration, however; the 
bi^op aM. "Why?" And I am a little bit concerned with *e pw- 
• dbility that we are now using the term "exercise" without too much look 
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DISCUSSION 

^ It whtfti are we ^dng the exerci^s to do. It Beems to me that the ap- ' ' 
i^opri^teoess of vrtany of th , 
boW;to yn^ to write test items, depends.on what you 

: intend tcf do with the W8^^ The ^libMr'V depends bii **what,*' ai^ , 

^im^i^^t^pf m S^ ^b&^wl ^ - - -^v - \ ' - - ^--^^ -- - " - ' ; - ^ - . . 

^^CHAmMAI^;^IGHtSTONE^^^ any mem^er^ the^ panel 

! wish to r^pond to t^^ - 

■ ^iCHAjRlV^ We accept the truth of this atdte- ^> 

■^:xami^, ^ \ ' ^' . ■: * ' > r V 

^ I wpnt to exprew own thanks to the panel meinbers whd imye 
presented an ixcellent r^ew of exercise-writing in humanities, in sciene^j 
and in the social studiei. When you see these papers printed in the pro- * . 
ce^nA,;7 am sure Uiat you will be ablia*^b study them much more 
carefuUy than ia possible in aq oral presentation I have had the oppor- 
tunity to see tbem,.^at leafet in mlftieo^aph fprm,; before the meeting. I ^ 
want to thank each of the speakers perionally,^ and on behalf of the^ 

^group for his presentation/ > . , ^ 
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Prediction o£ Educational and 
Vocltional Succ&s Through 
Interest Measurement ^ 



EDWARD K\ S^TRONG, *r 



^ What is success? What are' interests? How are they related? / 

Iq the*Hays when tests were just beginning to be iise^ Edward ^ 
Thomdike preachedlrtiat a thing exists, it can b§ measured.V Today 
wheii there are tests galore, he would probably ^Say, be sure you,^ow 
what you are trying to mfeasdre and then use only the t^it whi^i is 
specifically desigri^ tff m€5Sure tha* characteristic. ^ , ^ 

t Your assignment calls upon me to discuss the use of a test which was 
not deiigned to measure success. Consequently the conelusior^s what . 
should be expected^amely, interest tests do not correlate to ,|ny , 
.practical degree with measures of success. But I am convinced that 
interests have a real relationship witli satisfaction. ^' 
* ikying answered the a^ignniaut given rne I might be justified if 1 
sat do^i but I ani afraid your program commitfc^ would not feel that 
I had earned myluhchGon. Intend I am going to u ilHze this opportunity ^ 
to outline some of the things we know about success, e^nd ilileAsts and 
to hint at some of the things we do not know% We caift survey the 
whole situatioii for we donH know enough to (^prehend the^tal 
situation, Thtfre are hundreds of picccMn our puzzb-^pieces ^ 

mentioned in hundreds of artic Icb and bucjks--a^ tliere are lin^btqdly ■ 
many pieces that are missing. But maybe vye can obtain some gHmmeriiir 
of the complete picture Nv^ichnvill be disclosed some day whe^^a||the 
pieces have been assembled. Since my tirnc> is liniitfid. only those pieces 
can be mentioned which partain t%succcMs, iiiU^rests and I lie relation^p 
between tlie two. And the pieces that are nientioned must be described 
in a few words without refereucc to all the ifH,%nds, and buts tlmty^dght 
to be recognised. 

J %itrreHH =— Salisfadion ' ^ -'m 

What is fLant by succ'css? Sut'ceHS is an e^luation by one ur more 
judges as to how well some activity Is perfonned in term^ of a standard 
or criterion. There are thousands of activities. One niay be a success in 
one activity . and mi in another. For each activity there are one or more 
recognized staiidards by' which success is measured. These standards 
range from very definite objective muasuicsfo very vague suly^tive 
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rneasurei^Becaiiie several stuiidards aru recognized, an author, for 
fexanrpl|, may be judged sucoassful in terms of incomd, judginent of 
critics, or Nobel A^ward, ^milnrly a fireshnian may be called a syccess 
if he has an A grade a^ragc or even if he has merely passed the course. 
Fu^hermore, men arti rate^gp successful or not in terms of a given 
pQgulation ; thus %n athlete ^ffy be rated the bcHt high jumper in high 
8^ool,.ft coll^j^, d\\ the Pacific Gy^ist, or in the \forld. 
^ MfP are a^laimed as succ^sful and will continue to be so designated 
in everydayTOfeK But the basl|, for sucl^awards will vary from very' 
objective to hi^^ subjective^ stanchifds and* will^ differ appreciably 
^moBg differerit obsery^^. In any fiareful iiivestigation there shotfld be 
substituted for tlie vag^ftferrn of .success a definite statcmeiit as to what 
speciOc sfctivity is being ^nsldered, how well it is performed in terms 
of a sgficilic Btandar-etof criterion^ who are the judgt^s, aiid withiii what 
hpecific population the comeslants belong, 

fThere is anSlher cftmplicotion that^ muddies up our thinking. Every 
man is influenced to some degree by the evaluation of lus beluivior by 
others. But Ho a large degree ln!s behavior is detcjrmiiied by his own. 
eval^atfonB. These' twcj systems of evaluation lead, respect ively» to 
suq^ess or sa t isfijc t ion, WJ i a jt i h e y i s sa t i sf ac ti on P Sat i sfiu; t i o n i s a m ode 
consciousness accompany in»^ ac^lvilit's by wliit h one attains Ills own 
personal goals. Succg|s is based on sonf^j standard reeogniz^'d by others^^ 
lion is one's own appreciation that he has reuchexl his goal or 
advancing tf^ard his goalT Failure in tl^se respects arouses dissatis- 
aeJ,ion, AJ^behavior whether contemplated or overt brings some degree 
of satlsfaction-dissat^i^ctioih Uul only a few activities are successful 
irt tl^e ei^es of others. There is so rue rclationsMp betwefih satigjaction 
and BUjfecesp, but how mu^ is in»d(*t(*rnii h^il If one's goaf is to win the 
half qyle then suct'css is iq^t^eHHary lur'^utisl\j?'tiun. However a vast 
nuiftber of activities are tuu'ried on that mus^be judgecy as far from 
8u cc essf u 1 i iv t h 5 ey es o f o 1 1 1 v vA) i^l that n e v e r fc I le I ess b ri 1 1 g sa t i s fa c I i o n , 
I, for example^ lik^ishiifl^, grjlf aiyj gardening but I have never won a 
ipriz<|^ I simply enjoy Ihc'ni, obtain^satisfaeticm thereby. Brayfield and 
Creckett (2) import ver^^liti le 'fMalionship between employee perform- 
ance ariQ^CTnployee satisfawon, as expressAl by irttitude suryeys. Im- 
proved procedures maf: raise the IgW eorreyitions soniewhat. It is doubt- 
ful, 'however , it^herc is ai^ high n^atjonsl lip, since only n r|iQdicum of 
siiccess is necqssarf' to hol^nmst job^. It mustj^^ recognized/jihat many 
students and- employees do not strive foi' great success in the classroflni 
.Q^ ,^ori t||,e j ob . Th ey^tl oi i ' t w a 1 1 1 t cr^w o r k I ha la rd . 1^ t o ii i-goi n e o t h er 
l^ivity they may expenc^i great deal of ellkjrt and energy. They want 
to. en joy Hfe with their ftunilies in thelf homes, to have leisure time and 
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enjoy it in a n^llitude of wnys. other .words, their interests help 
' deterrainS the amount of energy they are wiUing to ptit forth and how 
hard they are willing to work, and sometimes these interests find no 
outlet on the job. After all only a very few can be rated as real successes 
in the claisroom or on the job but all may obtain satisfaction elsewhere. 

How is success achieved Success is achieved through the use of abilities 
and motivations. M,any dilTerent capabilities are utilized in perforfrnng 
any specific activity. Motivation is complex, involving ambition, willing- 
ness to work, willingness to forego present pleasures for future gain, liking 
to engage in the necessary: parts of the total activity, rewards of all 
sorts, etc. But if the competition is t^ keen or one disUkes what has to 
be done and finds too little satisfaction in it, one quits and looks fBr 
a situation, where there is less competition or for other activities more 
to one's liking, AU' through life everyone is faced with the problem of 
which way to go, in which directieyi one can reach his goals with the 
least effort and wjth the most satisfaction. 

What are the crilefia of success? In the busitiess world men recelvo 
preferment, increase in paf , aod promotion at the hands of their super- 
..visors. Many believe that supervisor's judgments are largely subjective, 
'usually biased, sometimes unfair. Such opinions, or ratings, are criticised 
because they do not truly reflect amount of work performed since it is 
assumtd by the worker that vocational success is a matter of p^duc- 
tion. It if quite possible, howbver, that many so called biase* opinions 
reflect, a feetter estimate of the man's usefulness than does his available 
produclio^ecord. There is probably no more troublesome problem than 
the deiermi\iation of adequate critetia of success in managerial positions 
and other complex jobs, such as selling. It is doubtful if there is a single 
case wlure dxperts have unanimbuslyjasreed upon tjie standards ui terms 
of whicl success is to bo appraised. In the case of many jobs in the 
office and shop the amount of work is determined, as &i an assembly 
line. Willingness to do the monotonous task must be far more important 
than abiUty to do the work. As long as we think of success as amount of 
production we correlate test scores with' sopie sort of production record 
and say we are measuring success. > ' . , 

Do we really know the criteria of scholastic success? The typical 
> answer is mastery of knowledge. The instructor's job is to imparl certain 
■ knowledge and to determine how well his students have acqmred that 
'knowledge. And it is the student's job to acquire 'this knowledge. 
Acquired knowlfedgo is measured by grades. Then wo claim that grades 
' meliure success and acclaim students as successes, or failures on the 
, basis of theb graces witiiout much concern ,p8 to the lack of agreemenl 
totween io-called. success in school and so-called .success in later life. 
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Ho# valid grades are Is Still a burning question. The validity must v^y 
greatly among instructors and among educational institutions. 

But the problem of what are the proper criteria of scholastic success 
goes far beyond such considerations. Here is involved the question: 
What is the purposi of education? Should the emphasis be upon acquired 
knowledge, or proficiency in its use? Knowing something does not 
guarantee it will be used. Or sliould the emphasii be upon appreciation^ 
enjoyment throughout life? It is often said that the purpose of educa- 
tion is to teach studentyto think, instructors ftnow whetl^r students 
have inpeased their capacity to think? Does anyone know how to meas- 
ure increase in capacity to think? Do we not want students to do more 
than to think; do we not want them to become extited, to bubble over 
with enthusiasm, and to imagine possibilities of using the material in 
unheard of ways? How much of our instruction is designed to propel 
the student out beyond Mie frontier of our own knowledge? And how in 
the world could an examination be constructed to test, the students' 
newly aroused imagination? It has been said tliat the country needs 
more leaders. But we munt not confine the meaning of tliis term to 
leaders along the accepted trails, we need those wlio will seek out and 
blaze new trails in science, philosophy and tlie like, creative leaders 
as well as executives. 

When emphasis is put upon grades as the measure of success then 
almost inevitably emphasis is put upon the students who get good 
grades and little or no concern is sliown about the students who are 
initially rejected fpr adniksion or later on flunked out. Should noli the 
emphaBis iie directed toward preparing all men as far as possible for 
positions where they will be reasonably successful and satisfied? Here 
tliere is need to consider specific abilities, interests, motives, aspirations, 
goals— in fact, the whole area of values. When all this is appreciated it 
is apparent that there cannot be one standard or measure of success 
but many definitions of success appropriate to all the diverse people 
who carry on the myriad uctivities that our country must liavu per- 
formed. 

What has been ^aid so far might be summarized as follows. First, a 
inan is successful according as society values hfs behavior; lie is satisfied 
accorljng as he l^imsclf values his own bohavior. Second, each man 
seeks varying degrees of both success and sat^faction, constantly asking 
himself the questions: "Am I doing the right tiling? Sliould I change? 
What should I do next?" The primary objective of education should be 
the orientation of each student rather than the acquisition of knowledge. 
But I do not want to give the impression that I am belittling the ac- 
quisition of knowledge— certain knowledge* in essential for success and 
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alisfaction in each activity. Third, we have no clean cut, uuaiiimously 
agreed uporii criteria of schola|tic or vocational success. 

Inlerests 

et us now inquire as to the nature of interests and what possible 
relationrfiips may exist between them find success. 

Interests pertain to the liking-diiliking of activities. Pleasantness, 
liking, belief, satisfaction arc affective states of mind indicative ofj 
^acceptance. vUnpleasaiUness, disliking, disbelief, dissatisfaction are in#^ 
dicative of rejection. The terms pleasantness, ^^ing and satisfaction 
are not clearly distinguished in every day usage. Our working definition 
is to use pleasantness-unpleasantness with reference to sensations, 
,Uking-disliking with reference to activities, belief-disbelief with reference 
to attitudes, and satisfactien-dissatisfaction with reference^ to feehngs 
and eiriotions related to the approach to or arrival at goals. Interest is 
an aspect of consciousness similar to satisfaction but not as thorough- 
going an affective state as satisfaction. 

What interests are possessed can only be ascerta^ied by noting the 
activities that are liked or disliked. To say a man ha| musical 
interest merely means that he likes more activities relating to music 
than activities relating to other groupings of activities. ^ 

To some degree interests arc indicators of abilities, not aff indicators 
of amount of abiUties but of which abilities are preferred. Thus a boy 
who is poorer in Hnguistic activities than in mathematics will seek an 
environment where he can use matheinatius rather than Englisli. 
Similarly he seeks a college major and later on an occupation which 
involve more activities he likes than he dislikes. 

How are interests measured? No one has so far derived a method of 
measuring amount of interest. I am very doubtful if tliere is such a 
thingias amount of musical, or mechanical, or carnpiug=out interest, 
except in thte sense that amount means tiie number of liked activities 
which are classified under some rubric. Instead, scores on interest in- 
ventories give the differeniies in interests between tliis and that. 

Three methods of measuring such differdiices in interests may be men- 
tioned. Kuder has grouped interests into certain classes or rubrics, sutjii 
as scientiflc, musical and fco^putationak The leatee must select which 
one among three rubrics in a triad he prefers. If ho prefers one rubric 
to all the rest he can obtain a very high, score but if his interests are 
more widespread he must necessarily obttiu lowgr scores on each of 
his preferred rubrics. The Kuder d(Jel not !ften measure absolute amount 
of interest in each rubric but relative amount among rubrics. 

A second ria^thod of measuring interests li that usred with the occupa- 
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tional interest scores on my mventpry. Here the interesti of the average 
meinber of an occupation are contrasted witli the interests of the 
average man, . . - ' , 

The third methcd of measuring interests is ^ployed with the intereat 
maturity, masculinity-femininity and occupational Hevel scales where 
the interests of men are diflferehtiatM from tfee interests of boys; simi- 
larly the interests of men are differentiated from the interests of * 
women. Scales of this type ^aye beien constructed in which the interests 
of the best students contrasted with the interests of , the poorest 
students and aimilariy the interests of euperior epiployees with the 
interests of inferior eraployaes. ^ * 

Measurea of success can be apprdpriately conipared with s^res of 
superior-inferior interest scales providrng the two purport to meaBur^ 
related abilities and interests* But measures of success cannot b^ apr 
propriately compared with scores dn the Kuder or my ocoupational 
interest Males since these do not*ptirport to express differences in InT 
terests between superior^aM inferior persons. 

It is a mistake to assume that scores on occupational scato measure 
amount of interest/Such scores indicate extent of agreement of the 
man's interests with the interests of the occupational group in contrast 
to the interests of men in genePah Figuratively, ^tlib scoriea indicate 
dir^tion to go. The higher the score the greater is thie certainty that the. 
individual should go in that directiort Direction to go is M^ell portrayed 
by our Interest Global Chart where lawyers are located at the north 
pole, carpenters at the south pole, mirtistera far to the east and prefeidAifs . 
of a manufacturing concern far to ifse west, rrjitjerest scories itidicate 
direction to go^ toward lawyer not .carpenter, toward ministef Bot 
president. Interest tests are not pr^iso^ instruments like a cbmpaSa* 
They can otily indicate general direction! It is a serious matter if ^fi, man 
plans to b^ome an engineer when his ir^tfer^Jts are those of a raiiiater^ 
or social/ science teacher, for here the diyfectibns are largely reVer8^|^ 
seems iocredible that a man should so? misunderstand himself. But 
have mys^f counselled'people who wesreintfent oty careers not m harmon; 
with their ^feaql records, their* Jr^tereitl or at tii^ea plain conrtmonspni 

U.m of Jntereats TesU 

- -J. - J . ^ ^ ^ 

us repeat tt/at no relationship can ever)be;detern0ned betwee 
interests and sug^ss Because sup^ss ii a^rief e abstraction. All that ca 
be dpae is^tS^rrelate intereft scores ^itli flieasuris of lome criterion 
rermsraf which an activity is judged. Consider urst thSrelatioriship 
fetweenliritereat #co#es and grades since grades are usual 
fas in^siireB of icholastic V 
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Ability and aptitude tests are useful in predicting what men can dp, 
Of better what they cannot do^ but they are not very useful in prediQtlng^^^^||> 
what activiti^ a man will select, I still remember the college girl w^p#p^^^ 
majored in physici with a straight A record who refuied every indud^^^^. 
raent to continue in grfidu&te school. No, she wanted^to marry and ha\^^^v 
a family. Interest tests function in reverie naanner. Tliey do not pre 
measures of inteUigence, grades and achievement but they do pee 
the dixjction nien Will ^o. 

Th^ are some exceptions tq this sweeping, over-all summary. 
example, significant coefflcients in the tliirties usually result when sec 
on the psychologist interest scale are correlated with intelligence ti| 
scoi;e9 and with scholastic grades! Such correlations are proof, thatil' 
ar^ Jhterest items which so faction. It is possible that if enough'^^ 
itetns were utilized a scale might be developed which would corf^la 
substantially with such measures. This is all the more possible. ^|^e; 
test- scores expressed interest in some ireitricted area of knowli ' 
vtete correlated with scholarship in courses similarly restricted 
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Jft is possibfe that there is actually a ^higher relationshi^S^^ 
iiiterest and achievement scores than so far repqrtefl J One rS^n 
iuch a st^em^t is that in many cases app^priate interest sc6g» ii 
.^^^^y restricted in range yielding low^Arrelations. . Thercr 
\ v^^tovple, mafty students with high engine^ interest 
\''jiey(E|f studied engineering^t there are few sftidents^ with low engiii^r ^ 
irilerest scores who^ha^taken moje than one engineering coursie; W0|^ 
^ Tft^fit j^ect j^i^er correlations among students in requiresMour^^^p 
^ where .Mnistud^ts ^not possess interests appropriate to Uf^i^i^itef**'^ 
"l0hje^&ich^rivei^igati<|^oncerned s were oi^er^^^^^ 

H artrt;^# feker^ pBrSOmel-psychology course which most^j^^^^pii ' 
, nbtv^ht^4)l Herfthe data showed that interests mtt^ 
^ - , wi^Kcoutse graded as did intelligence test scores, f ' 
* . Abotit the same telktionship holds between interests aii^^Mpational 
succe^ & betweeri irjtcrests a^^^ademic success. Most,|phe researcii 
.yi^s. been^^poncernetf with fAnien; salesnicn, productil^ and sales 
'managers, and resoatcH crigineers.^orrolations betw^n some criterion 
of iuc&ss atkl oocupational^fiterest bco^ hivye boen jbr the most part 
^.negligible. Vmm la one strittmg jjixcepl^K Smres life insurance, 

rl '^ealtot, anri^le^ ma^Wager scales have correlated 3^^|c^itly with sales 
^ pMubtion ariyd negalively witli turnovqr among i^^^oups of sales- 
Trt^m l| is qtiite'jpoisible that tliis rekitionship Syt^d hold tor other 
s^^i^efa if tti^r^^^e scales which mpfe^^iily represei|te!| their sales' 
Mii:li*iV^ fvu^ i.iSQiS.. >vr i-HineiV^8oales may po3si^|^ explained 
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on the premise that Balcsmun mml hv really iiiterewlod in their work if 
they are to succeed. It is posBlble that otlicr octuipational scales might 
correlate significantly with nieasures of success if adequate measiiros 
of success were available. 

Occupational interest scales were not designed to predict success. If 
it is desired to predict success then iiiterest scales should be developed 
contrasting the interests of the best employees with^those of the poorest 
men. Research with small samples suggest the feaslbiUty of such an 
^approach. But actually, it is difficult to obtain lurge samples of inferior 
men. Although secondary school administrators complain about un- 
satisfactory teachers, a serious effort by Gilbert Wrenn and myself to 
obtain the names of such teachers made clear that higli school principals 
in California would or could iiot supply more tluui 62 names. It is to be 
hoped that the Life Insurance Agency Management Association may 
attempt such a scale. They, if anyone, cati secure the iiamcs of many 
unsuccessful salesmen, 

A more feasible program would be to contrast the upper twenty-five ^ 
pep cent of an occupational gmup with men in general on tlie assumption 
tKat the average man is no more fitted to be a success iji that occupation 
than those who have been found wantiiig after a tryout. This would be 
equivalent to occupational scales based not on llie average member cif 
an occupation but on tlie superior members of tlie occupation. Such an. 
innovation might be more uHcfnl in Helection but it is dnirbtful if it is 
desirable for guidance. 

Turn now from the relationship between inlerest,s and success to ihv 
relationship between in teres ts and orientation of students and employees. 

There is no question but that oerupaUonal interest scales difTerentialc 
occupational groups. But do occupational interest scales appropriately 
diirereutiate students and what validity c^scoros on these scales possess 
in tilt sense of predicting the cjCcnpationH nj wljicli students will eventu- 
iilly engageP 

Although my test was luit desigried to dilTerentiate between college 
curficula or majors, Berdie (I) reporls that interest scores predict such 
^.choices better than either aptitude or achievement tests. This must be 
^tecanse men. choose curricula which they deem appropriate to their 

occupational choices, or vice versa, 
y Graduate medical and business students at Stanford are very clearly 
differentiated by physician interest scores, Sevetity-tliree per cent of 
medical students obtained an A rating in contrast to one per cent of busi^ 
nessstudentsandonly one per cent of medical students had a C rating in 
contrast to 83 per .cent of business students. It is doubtful if these two 
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gruups could have been diirereniiated to any practical degree on the 
basis of aptitude test scoreg and undergraduate scholarship records. 
Occupational interest scores not ordy differentiate occupational groupB 
but they predict the occupations men will be engaged in later on. The. 
data from an 18-year follow-up of Stanford students show that if a 
student had an A rating on any one of 16 scales the chances were 78 to 
22 he would actually be engaged in that specific occupation 18 years 
later and that if he had a C rating the chances were 83 "to 17 he would 
not be io employed (5). The expectancies are really better than the 
above figures indicate since all men employed in a closely related occu- 
pation were counted as misses. Thus all men witli an A rating on the 
accountant scale who were engaged in office, credit or purcliasing work ^ 
or even as CPA's were not considered to be accountants. So far np 
satisfactory method has been discovered whereby credit can be given 
to those who entered occupations closely related to the occupation in 
which they were tested. This is unfortunate since high scores on an 
occupational scale should always b© interpreted as favorable to related 
occupations. 

Occupational interest scales have bqen devised primarily to dif^ 
ferentiate between gross occupalionM^ groupings, as, for example, 
chemistg, engineerSp physicists and niathematicians. Recently it has 
been demonstrated tliat sub-groups of engineers^ of ^psychologists 
of medical men can be similarly difrerentiated on the basis of interests?^ 
Several hundred medical students have been scored on the four medical 
specialty scales and a ten^ear follow-up is planned in ordfcr to determine 
how well such fine discriminations predict the specialties in which 
medical students eventually engage (6). 

There are many orientation problems facing young people. Interests 
must play a role in many of them. Which students should eiiter the 
unskilkd trades, which the genoj-al run of buHiness activities, and wliich 
pursue graduate work are not questiofm pertaining merely to abiHty. 
Many competent high school graduateB do not go t<j college because 
they don't want to; many collegn men enter ocoupalions of lower status 
than their IQ would predict. 

There are certainly many men who really enjoy their work and they 
are to be found in possibly every occupatioih I once employed a gardener 
with an IQ of not over 80, whose eyes would light np with pleasure when 
Jt^asked him to spade a tough area or to dig up an old stump. We dun't 
know, however, whether all me^ are so constituted that they can enjoy 
appropriate work; also whether the number and variety of jobs are pro- 
l^rtional to the number of men witli corresponding occupational 
interests. 

BO \ 
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We don' I know liow^nmny adults read books for pleasure whu never 
wcn^t to high school;%e doiiH know how much High school and college 
has ificreased reading for pleasure, or even decreased such enjoyment. 
And we don't know whether those who read for pleasure differ in 
abilities or iriterests or Both from those who don*t read books. Further- 
more, we ^on't^know to what extent English can be taught so that 
people will read for pleasure and whether such insttuction^can influence afl 
sludents or only a few, and if hut a few, how they differ froni the rmL 

Several law and medical faculties are worried about certain alumni 
who are interested in making money, or are engaged in non-ethical 
practices. They want to know how such future alumni can be identifled 
and eliminated. It is doubtful if intelligence tests and grades would 
diUerentiate such students to ttny worthwhile degree. 

These ^amples /are only samples of many problems regarding the 
differentiation of young people, of aiding all young people .to find a 
career in which they can be reasonably successful and sa^j(ie4| 

It is generally assumed that people with high IQ can succeld in any 
career; consequently it is of niinor importance what career they espouse. 
Tins is largely true, such people can so perforni; but should they,^ A 
considerable number of men in Xhm fortieg and fifties^ have sought my 
counsel They had ccane^ to realize that their work was work, dreary 
work, and they were tired of it; and liad discovered that some of their 
friends found their work\interesting, exciting. They , reminded me of 
driving an auto with the liand brake pulled back, when one is wasting 
gasoline and wearing out the engnie and brakes. These men were tired/ 
bewildered and oneVas on the verge of a serious mental breakdown! 
They all scored low on interest scales appropriate to their work. The 
objective of my research during 33 years hits been not merely to direct 
young people into work they will enjoy but to prevent older people 
from wearing themselves out* doing work they don't Hke and being 
forced to continue therein because in most cases it is now too late for 
them to enter a new career. The present da^ interest in retirement and 
the appariing number of mental cases in our hospitals appear to be to 
some degree the resultant of people forced to work at that which is 
foreign to their interests. 

May we repeat in summary that s^uccess is obtained by skill, knowledge 
and drive. One's interests are not particularly relrted to how mucli skill 
or knowledge one is capable of acquiring but they are closely related to 
the drive t|iat stimulates orfe to make use pf all of his , capacities. It is 
in the orientation of students and employees toward their most pro- 
ductive activity that int^esta play a most important role. 
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Ever since the beginnirig of the use of standardized^tests, specialists 
in the field of measPfcriieiit have had a good deal of free advice for lejt 
users. They nave admonished useri, to choose tests whicli are designed 
tqEineasure tAs major eduoational objeclJves of the schools, to adrninister 
these tests with faithful attention to the standard dir^tions, to employ, 
scoring procedures that will assure accuracy in the scores, and to in- 
terpret the results in the light of statistical data on the renability and 
validity of the measures. Much has been said and written cohcerning 
the "education of schools in tlie hheaning Bj\d use of test results." 
% At the sanie time, test specialists have, in recent years,, become in- 
>creasingly aware of the need for communicatioh between producers and 
consumers of standardized testa to be a mutually active proeei§. They ' 
have learned' that teachers, counselors, and school administrators have 
much to contribute* to test production. Many of them have, in fact,; 
come to recognize tfet school personnel must take a leading place ifi} 
determining the objeotiveBj scope, and content of the tests which are 
used with studeftts. So 4he current trend is toward the developinent of 
new tests as a' team effort by administrators, teacherl, counselors, and 
test specialists working in close cooperallon. 

This afternoon's sdssibn is in line with tlie modern desire of test 
speciaUsts to learn about measurcnient needs^f^onl test consumers. First, 
we shall. hear from an administrative ofilcer of a large €ity school system 
who has himself made numerous contrlbutiona to measurement nnd re- 
search. Later, a national authority in the guidances field will present a 
couViselor's view of measurement needs. Each paper will he discussed 
by a leader in the field of educational mefls\irqjn<^iU, VVhilc; the papers 
are being presented, hi em hers of the andien^e^fiHf^^Jnvjled to note any 
questions or comfnents for use in genorat & the end of 

the session. ' ffi^^ ' 1/ ■ . ^ . 
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Tests BsA testinf are of contlhuing concern to schwl administrators, 
Ttis aftertioon I propose flrst to review the usts of tests from tlie point 
of view of the school administt^tor, and second to express some ^f the 
probleiris in the field of testing as seen, by the school administrator, in 
the hope that . ^ecIaUsts in testing may be able to help meet these 
jprctoleiM more ad^uatbly, . . ^ 

^ The Scf^l Adminkirator's View 
^ A general funption of school administration is to provide the facilities 
and means when&y instruction can be carried on. Tests constitute one 
such means. Pf^ably^ the primary use of tests, frogi the point of view 
of the school admimstrator, is" to fmlp teagfiers knom and andersiand 
chi^en and Ifmir meds kUer. In the last twenty-five years^jiere has 
bei a gfowing empharis on th^importance of having tochers not only 
know what there is to^know about child growth and deyelopment in 
general but also have control of procedures by which tft^ 
better acquamted with eadfi chfld's capacity, interests, antf'a^omplislj^-? 
meftts in every major field with whicli the school is conceriftd. In the 
Detroit school system last year, with a total membership of 2:75,000, 
^ ^ut 96,000 intefligendk tests and 1,300,000 priii ted achievement tests 
of one sort or another wer^' used. Of the total number of such achieve. 
, ment tests, approximately 1,240,000, or 96 per. c^it, making u^what we 
call the ' optional testing program, were ma^ available to teAcher| at' 
their request for use in their classrooms afeit^|ime and in the way they 
Wished/No reports of the results of these tesW were made centrally. The 
tests were provided solely to ^nable teachers to GompUe infc»:mation 
about each pupiFs status in relation to sohie instructional goaL The 
i^ther ^hase Of achievement testing Is the gu%nce and counseling 
tfe^lffg program. This consists^of the Iowa Multi-Level Test of Basic 
Skills and a locally developed inventory \Qf experiences and interests 
which are administered in grades Low 4, Low 6, and Low 8, ? 

A second use of testi, from the point of view of the schTOl adminis- 
trator,- is to enable teachers to emlmie meh the acQomplishment of 
pupils in partwuhr coupes. This has reference esjpepally to the secondary, 
pleval, where ^roifiotion takes place usua% ^ubject rather tb^n_by 
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entire grade; In most citif |, I believe^^t t^p are rarely^i^vier used 
as the sole criterion for pitlsfactory Cj|^^ltion of secondary school " 
courses, but standardUHd tests can be" and are ^being uifed, to affprd 
additional Information to teachers as to the prq^^ess made*by stud^ta^* 
iji particular courses. This use of tfests is even %ore irttpdrtant "^iWien 
jiifitruation is given by specirfmeant^such as correspondence or televi^ 
^BiiE« h^^j^^"4-^^--t4^^:-- 1: — — - - - — - ■ - * ^ ' ;- - -- 

A thij^d irripdrtknt use of tests is to help in the coiinseling and^0iid^)w^t 
of chUdf^milarid pariicularly in classifying them for imtrmiioT^t purposes. 
Tests both of capacity and achievement are needed for this puipose. Let 
me refer again to our Detroit experi^ce. The results of intelligence tests 
and the achievement test battery provide information which is helpful 
to counselors in the junior and senior high schools in guiding students 
into programs best suited for them. Such use is a major reason certain 
tests are given just before the end of the child's experience in the ele- 
mentary and the junior Jiigh schooL 

% Test results are use^ extensively in the classification of pupils^b|Jh * 
for assignment to particular courses and classes and for abUity groupmg 
Tirttilin a particular class. 'For example* the decision as ^ to whether, a 
nflto-grade student takes college preparatory algebra, general malhe- . 
ma tics, or arithmetic review depends ypy largely on the results of the 
tests that have beedff administered in tho eighth grade. Likewise! 
the sub-groups in' a second-grade class in reafing or an eighth^grade 
class in civics or a tvvelfth-gfade class in Englisli may te based on test 
results, ' 0 

A significant phase of tho guidance use of tests is as a resource in 
conferences with parents. The school that has a good background of 
testing data for pupils is prepared to. explj^ objectively to parents tlie 
reasons for proposing part^ular courses of action. In this connection* 
the Detroit achieveinent profiles of children at the fourth-, sixtK-, and 
eighth-grade levels give considerable information. 

A fourth use of^6^.^, from the adniiiiistratdr's standpoint, is to an^ii^ar 
ilie question of how well the school or school system is doing in iris tract ion 
generallyiMnd to aid in the rediredion of emphasis in the instrujcHonal 
program. Members of boards of educaUon and citizensV advisory com- 
mittees continually ask how tlie test rfelults in the local school systein . ^ 
compare with those in other SQhool systems. This is a very real question^*' 
and one that is exceedingly difTicult to ans^er^ fully anci accura,tely. 
Truly comparable data are diincult to secure. Pupil populations 
different school systems vary^widelyi grade for grade, in' general ci^ 
pacityi in home environment, irj age, in time aJlotnientSr and mdeed^r 
in every factor that" affects acadernic success. Interpretation of dtf- .a 

. - ^ ■ - ' ^ 1% ' ■ ' " ' 
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feronccB in teat rosulli between two schpoh systems calls for great care. 
Furthermore, wheii the comparisons are unfavorablo to the local soho^^ 
ivstem, there is a tendency" t6 withhold thjs lirforisation from t» ■ 
^generaf public. The difficulties In dealing witff the question, liowevor, 
do not mAe it any the lew important. The Buperintcndant of schools who , 
is chirged wHh the gBneral oversoeini of the entire school program ne^ds 
information re gulBriy ^o t he attainmen t at the various gradeJevplB^ , 
in comparison with attainment in coppaFSBle Bchools elsewhere. 

Knowledge of the comparative results jr^arious fields of mstruction\; 
can afd the administrator materially i| redirecting cniphaBis on an m- 
telligent basis. For example, if the test results indicate that at the end 
of th«»filementary Bohool peribd the schoel system is ac«omplishmg good 
resultslSKnparison with other sohpol iystems in the fieldi of rpdmg 
and spellihg^t lit* in arithmetitf or akiUs in social studies such as mup 
reading a shVt of ' emphasis tnay be ind^nated=eilher by a diange n. 
the time allowance w in the selection of materials, or by some other 
means. The problefc of reUable and, truly rcprescntativq norms is 61 
great importance in this use of tests. * " i l • 

A fifth use of tests is to anmver the qiieslion of how llie resulls being 
secured now compare with the remills ,mired Jlre or Iwmily vr Jijtj, yms. 
ago This poiuj is raised poriodicnHy by people in llio commuw^. Often • 
the asaumptiSi is that children do not learn as much affthoy did X years ^ 
a«0 Whether, or not this is so is a" question which every school ad- 
rrtinialrator would like to be able to answer satisfnclonly. Praclicany, ; 
v hb^evep, this is aft exceedingly difficull question to answer, because 
teat rfaults of the past usually-arc not avnilablo. Or, if .such niformation 
it available, the presumably comparable tcRts of today have changed 
content and may not be truly comparable. Or emphases m iiis«tuclion 
have changed-, no tests were given at an earlier perig to measure 
attairiment of objectives considered imp«rtaiU no*, ana*consequenlly 
comparisons are incomplete. Thus, it seems that test makers would do 
a great service to school administrators by providing tlie results ofjhen- 
and-now tertt, where these are determined in school systems able to 
carry out fadh studies.- But care must be exorcised to insure that the 
pupil populations then and Ww have similar charactoristica m terms 
■ of general social and Jntellectual^evel. Compansons between sub- 
atantially bhanged pupil' populations may rende^a disaerv.ee to the 

profession generaUy. , , . » 'i- „j 

A sixth use of tests .in school systems is to help in Ihe seleclwn and 
prmmUon of teachers and other school employees. This is a relatively 
recent but rapidly growing development in many^ool systema, in- 
cluding Detroit. TeBt8.hap.iccomc an,important elwJient ui personnel 
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'ftdmiaistration. Ii^ Detroit we now useleats'as a part of the process in 
the, initial selection G^ee^l^ers, and in |!romotions to department head, 
counselor, and assistant principaL Tests a^e used in the selection and pro- 
motion of a wide variety of non-teaching enaploy^s, including i^nog- 
raphgr, clerk, bookkeeper. Janitor, cleaner, and 'lunchroom workw* 
There is nee4 for more vaUd ttnd .preHictive* instruments in all theie 
-■areaSi^-^^ --^..^ .v . - - - - 

viJ' . ProbkM in the TeBling Field 

Perhaps the central problem in the field of testing, from the ad- 
miniitratpr's viewpoint, is the reluctance of many tpacherli to use tests 
as fuU^ ap their value would appear tp warrant. What are the reasons 
that matpy teachere, and "some admlnistratdrs, have for not using tests 
nisively and^as fruitfully as seems dearable? . , 
Bt group of problems centers around the tinier eJJoH, money, and 
^Ip^fence involved in using iesis, Teac^iers and admlniitrators alike - 
ayj^ests which will result in maximum information with the minimum 
exjp^diture of time and eflPort and with the least inconyenience to school - 
staff ;ind school program* In this connection, one oft^n^expressed need * 
is for tests that' can be handled in one or more normal ciaM periods. 
Having an uninternmted test that requires over 40 minutes Imposes a /| 
problem at tlie jtmor and senior high school levjel because .the tra- 
ditional length qf'^e^tess period is only ^5 minutes. Tists are needed 
which can be a'flmlnisteredT in_ their entirety or in parts, in time units 
of no more than 40 minutes./" ^ ^ \ . ' 

Another point made by lechers is that tests should be provided i^i 
foripats which can be used comfef jably on the ^BmfUest working surface 
usually provided, since maBy classropms are egujgped wifcB tablet-arm 
chabs rather "ttian the older type of desks with large #iurfaees. Teaahers 
fld^that test booklets are more usable in such iituatioiii^^^beriQpage^ 
tt-^e turned,dfi4 folded under, thus taking- up a: mi^inglum oPwork 
ace, WitWbMl||ets tliat nave many pages, some type of'jrin^or spiral 
4dipgjs^.^wabl&^ ^ j 

/Teftcheri^^d^ adrupistrators alike prefer editions witlb scphrate 
answer sb^s #ora stapdprfit of economy, both in initial cost of 
the test^ahdfinswer iheets and in time and c^st of scoring.. * 

Adaptability to machine scoring is another point to be coikidered in 
insuring more extensive use in the schools. A year ago thjel Detroit 
. . schools adopted as the major test in their guidance and cpunieUng test ^ 
.^ ^-;^fe|Tam the ^ Iowa Multi-Level Test of Basic Skills becausej among 
^ v oifti^f reasons, mechanical scoring was availa|3le at reaHpnable ^ost and ' 
# ' 'iiin fast service. Tv^o times last year, answer sheets to the nuniber of 
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nearly .thirty thousand were sent from Detroit, scored, and returned 
with results irtthih three or four weeks. The fact that th^ actual scoring 
of these very loiig ^sts did not need to be done by'the individual teachers 
flgured greatly in the teachers* .willingness to use thfe tests and the 
Jisults ai fully as possible. > 

The use of t|je same test booKlets a numBei* 
construction and materials to resist to a m"' 
wear and tear of ten 9r fifteen admjiTiltratio' 



of limek calls for durable 
xinmm dfigtee tlie normal 



Another suggestion often made by teachers and ^^incbals is that 
the author should offer' extendefi supestioni for inteJ^^HpH^iort And use 
of the test results. Too often, from the.begiiirting of the testing move- 
ment in educationV there has been^a tendendy to give a teW, look at the 
results, and ttien file them with little if any consequent action; Today 
the situation is mueWmproved in this regard, but the test^iaker still 
has an important responsftility to report; to f rospective^ test ^ers what 
other test users have found helpful in utilizipg test results # greatest 
advantage. Such suggestions may be included in the test manual or in 
special supplenientary bulletins devote^ exclusively to wa^f following* 
up and using the test results. ^ ' ^ V ^ 

A second major prdblem that administrator see with regard to tests ^ 
is the med for units tjiat are comparable fromfleld io field and frm Qradi^ 
lo grade. This problem is not a new quc to the specialists iri te|ting. Y|)u>^t 
have wrestled with it for many year§. You r|por|. test results in grade j 
levels, in various kiijds of sojalled standard scores, in percent^s, * 
in other ways. It is obyipui, of course^hat whatcv^ unit^is used has* . 
meaning only in terms^f the a^quacy of the sjni^e^of^e population 
on whicli the norms are based. The question to whicl^ ^tsh to addres^ 
attention here, hl^eyeif is the selectior^of the unit itself. I Ihink school 
administrators would be glad % have all achievgnienL test results in the 
elementary and secondary scho«s exprfssed in the s^e units, so that 
eomparisoiis maf be nAde ft om test to^est f* parflcular children at 
one time* and fo^successive admrnistra^jons fr^h grade to grade. I urge 
continued attention to this probftm of finding the unit that meets best 
the varW requiremenfc of a good unit Wi whicl^to express test results. 

A third proy^em that conjprns school administrators is ihc: need for 
norms ffml are Based ^ adeqmle anS^epresenl^m data for variom lypes 
of popmlion. For example, two diiTereiil reading teats are admimstered 
to High 1 pupils. Even thougli t^e tests appear, at least superficially, 
to measure tlie sarfte group^f reading abilities in subrtantially the same 
' waf , the results gn one teat may indicate tha* the pupils are a half-year * 
below the norm, while oi^the other test they may4pdicate that Uiey are 
a, half^y^ar above the n«rm. Situations of thk type give rise to grave 
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doubts in the minds of teachiers and schooLadmlniitralprs. This in. turn 
hindere the wider use df tests by whool personnel and the fuUer davelop- 
merrt^f the testing movement in scho^ ^ 

Th^e-are problemi alio in tiie ade^acy of norms in the ciase^of some 
teats. Several years ago in Detroit we tiwted seven thousand students 
iii^^Wgh school" subject wittftthf only test commerdally available that 
iee^l^ suited to our need. But the norms offeriBd by the author 
based on a population of fewer thw^ four hunlr^^^dmta i| 

In this connection I should note also that school i^inilifaio 
welcome differitttial norms fOTV^Iffarent types of ^hool popultftio^^ 
Again, let rtie^refer to^ experienqe in Detroit. Ours "is m tiighly Itv-^ 
dustrial city, with a changing population. Wp would like to have ribms 
available fg^thdtestfi we use whiqb would represent ayerage achievement 
in cities similar to Dfetroit, and possi^y #ven for type ire^ within large 
ci^l^ withw^ than avOTges for sta^s witbpreddmiflanyyVu^i po^I^ 
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tions or dWes characteristic's riiarledly 
Detroit* To repeal we need norms for every jtea. 

, sentative of 'Various types of populations. ' 
The fourth problem to which I want to direct att« 

^ achmerr^ni tests for every major coarse— espe^My at 
hrnL Thm ymr, fpr example, Detroit is partic^^.^ 
project* sponsored by the fund for the Advancement^^ 
designed to evalpale^the teaching of large classes by me^ 
\\(e are coij^uc^]^ the experiment in six subjects atl 
-levels. One of tlj^A is American literature in the seni 

vOilr instructionjtf tfesearch staflf checked carefully an^ ,^.^., 
find a standard ^et^jthat was judged to be satiaWct©ryjfrir^g^^^ 
in comparing risulta in classes in American literatuft us^^^^dM 

, vision course and thosatDonducled in fte ususfl manner. TCei^ i 
need foFlests in every ^lajor field of 
levels of maturity. 

In Detroit we have been m^ng 
arithmetic* and spelling for foi^y yeai 
usetf^different tests in the social sludi#?in 

of home economics and industrial arts; .^^.^i^^^.^,^^.., 
that we have only rarely used te5tsi^;^t^|&ik b^^ ^^l^ion, 
physical education, or science, althQi^jf''ihe^^i^^|cdn 
part of the curriculum in - - - ^ ^ ^ 
relatively fe# standard 
studi^|idthough these 
the i^^uction^l time 
one is, that we, ajt least, 




ition at each of thtl^difiei^nt 

i 



iive use of tests iiilttodm 
time to ;time 
igh,%nd in sonae phf^s 
f^^^hy of riotei^^o\^veri 




l^^p^l^^r€ce^t■ yeMs ^-^e^have usea 
3n^i6%-e^r^ai&ri. andift the social 



instrudtiM take/& ^u%fantial share of 
L there, varied resisons for thidif^t 
been a^S to locate tests that m^k to 
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j^-s^; in tlrtse .^tl^WBelitrf r . ■ . ' y, 

; V ■ ■ ; TM 4telWbl«P I ;want to mention is the needfyr W?-'o''S""Pi^*:: ' > ^; 
' t , ' ui^^M^ ^iil' TMosure morp comprehensively ilw mhmen0ii i^^ 

pupifesioujorrf dif i/ie bufco^^^ seefe Ihmugh the%mM of llie , : J* . 

^ ■ V and testiiS of Americap Uterature. The teac^^^ of AtMrictni literatui^ £r 

• hae a numbep of o ^ectiyea In mind. These include aW^|gan|e 51th 
; .. ': : Wfitfra and writirigs, oreation of thexlesire to read^merit^h .W^ 
i' " • aM cfevdlopment of taste and diacrimirmtibn in litew^vrP A, truly 
; ^ • ad^iate test of pupil growth in; Americaif lUeratuje. wuld^aiate 
■ ' : ' not- only the knowledge pupils have acqulced abou^ Aipef ican wfiters 

• ^ -j Aki their writings, but also the degree to which they have/develop^ . f ,; 

; ' J- ttote e(rid di^^ a»telt . 

* • :■ ; w^uld i»^vide some iddication bf the UkeUhood of 

.'^td continue reading American literatura after the cpufle .if odmpi^ted.. ■ 
V ' * ^TOs K what l-mean lj^ of tests-tlwt m^sure-mMp » ,., 

: / y ^n^reheimy^y^^i^^^^ towardvfho BchwlE^.totiil ' ■ ' r 

iP :;'^^M6upor^oye0tiyes.J', ^.^ . ; .'y / ^_^,.C . 

■ 'i-^ Oiie fijittl pcobfem. I want to' present you. Thil miiiA^omhip. ^ *r • 
P - A ^fsm^d-Uk^ toAhe: mjarks Ihal teacfiers assign pupils f of their ^vork .im 

' ' %'seh6otiu^ecli^:Bml markB. ahd report cards, ard.thq^suhjeets of ^ " ^ 
. ' cqnlinuing^ate practically every w 4 
■ • '■ ■ '■■k •ally ijd^ syslemB tht^.are ^ , , T 

P -i ■ ■ rwdy in' ilW^Jsiitew te^^ The pupills marK mfF Tf^PfO^ / ' - * 

■ ■ ' of feblHl^'W lomfe absoluy sc or Ins ability m relattbrt^thm 

' graders a Whole, or hid ability iivreW^^ % 

') i ■ '' v of ^iiirf^tlteae criteria ^ . ^ ,■ ■ ■ , I '■(; "' 0^ ' ' ■. ' ' 

- ; •'. ■ ' ; Is :it:' posaBle'f^ with praPtiMeltf fMs m^ . * 

• / v' - wKiChVobj«otive:t&^ sboros may be uscd to help rtiaki3^fl'raa^,ni2re' 

V V mtonW^ful? Giin the teat scores bo expressed in a tm t^inMi rnvm ^. ^ 

; ^ ■ . parerfts a'beUer idea of papj* achievement? Is there jperhaps.sfane , 

■ ^ ■ liiodern. adaptatibn of the oldiccd^plishmeut quotient whtchjna^ be^ # 
' ' : used to^ supplement the absolutd leVdl of acjiiovement revcal||;b^*tiio ; » 

standard (est. anti thus give some estimate of tlio studenVi a^niJjhsh-A. ^. 
' . ment in delation W his capacity to aohyro-either his ^ral cMacit^ , 

or his ^ebific capacity in a particular ^rea? CortamlypiB. proHgm m 

■ a reardne in all school systems. I stale it here ui the htJpe;tl]^l^^ 

■ gpeeiallBts in educational testing may bo abjo topffer some helj^n 



area to school administrators. - W i-M-v w ^ 

You will note that I have said nothing here about vahdityfrelidb^ty 
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^ival|I^pa of forms. School adminiptratorB rarely talk speciflcally 
t th^Jtechnicpl chgracteristiGS of g 
'Vv^i iittoe, to i^en^v does not betoken any lack of regard: for their 

^; ^ -im^brtphca Gertainly the good test must be yalid, ^nd must be reliable ; 
f-v ;\alt5^I^t9.^^§6^ms^m^ Be . equivalent. I l|avl^ rather attempted to -give 
^ %J^re9fi^n|to id uppernioit in th^ rainds of school 

>ri as th^y cediteder teeiing in relation to the schools in 
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Dr. Rankin, we of the tttt-making frate rnity are in^ed tp ypu for 
he^ini ui toJsee, thrbugh the eyes of the school administratis the 
functtons which our instruments serve in the schooli, and for your 
thoughtftil recommendations as t6 ways in whfch we may make these 
instruments' more useful. The problems which yo\i have outlined for ^ 
our oonsi^eriktion here this afternoon are, .1 know, by no means peculiar 
to Detroit, but reflect experienoeB common to school administrators 
across the aountrjf. X trust that I shall be able to reflect the reactions 
of the test producers as faithfnlly as you have reported the foelinp of 
school administrators. ■ i. , *<>, 

■ I am immediately impressed by the fact that tlfe problems ip which 
Dr. Rankin invites our attention as test-makers are not new pfroblems, 
but rather, ones that have been persistent , matters of concern for at 
least as many years as liiiave been associated with the testing field— 
now more than twenty. Can it be, I have asked myself, that the test- 
makers have been so unresponsive to the needs of test users that no , 
substantial headway has been made in dealing with these matters? 
How vaUd are the reasons which we have traditionary advanced for > 
being unable to' comply more fully with such requests as Dr. Rankin 
makes of us? What are the prospects that we shall, in the near or more 
remote future, be able to give Dr. Rankin and his fellow school ad- 
minlstrators some of the help he has here asked for? 

Titm, Effort, Money 

■ Dr. Rankin first called for 'improvement with respect to what we 
may term the" engiheering aspects of tests— physical chuactBristics, ' 
time requirements, simplicity of handling, machine scorability, and the : 
like. Test makers would claim, l am sure, that they have been acutely^ 
conscious of such considerations; that they have lyade notable steides 
during the past twenty years in the directions Dr. Ranl^ deems desir- - 
able; and that, if some tests now being produced fall shCrt ^ what the 
consomer would consider ideal with respect to these cl^cterirtics, it 

. is not because publishers underestimate thetf importance, .^H^-^^"^ 
of the many other factoVs which must also play a part 
Let me be speoifle. There has been a decided effort«^^; trfte, or 
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work units of them, tb the 40-minute liml^^r/Rankin proposes. But 
cdnBiderations of rfUabllity and yaliditv^j^etihaes argue for a longer 
^t, whereupon a pradtical d^isioir^^^s to be made as to the list 
Qompromise between admimstrative^onven^^^ and adequate meaiure- 
tnent* 1^at is the t^t^maker tb do when he discovers that to prt>duce 
an aaoeptal^ly reliable na-eslsure in a given area requires a test of some: 
SO or 55 minutes duration? The cdn^ientious test^rnaker will not 
Idiffi^fci rlfte^l re^f^steying^gdthi Ifaitr 
nor does he want to extend the test from 50 to 70 or 80 minutes merely 
for sake of i^ng:two clara periods. The time required for a test is, 
aftar all, lairgily a function of the complexity and scope of the ability 
or abilities l^eing measured, and is not entirely to be set at the discretion 
of the teit maker. , 

As .to formaty^ it is not bappanstance that sq fn^ny tpts fire printed 
as BJ^ % il bbokitts* Coniid^rations of eflPectivJfe utilization of page area/^ 
attractive^up and cliu^ity of page layout, and size of dheet which can 
be handle^economically on available presseSi favor the 8J4 x IJl pagft 
size* or^^ne fairly close to it. When a booklet of this page size is used 
with an answer Aeet, whether of the IBM or Iowa variety, the total 
area covered by booklet and aiiswer sheet necessarily exceeds that of 
the tablet-arm chair of whicK Dr. Rankin speaks. The cumbersomeness 
of inaterials appears to be an inconvenienco thtit most students, at 
least at the high school and college leveU quickly learn to cope with. 

\ ^^^\^se of Separate Answer Sheets ' s 

The matter of adaptability of tests to machine scoring, through use 
of separate answer sheets, continues to be deserving of the most careful 
scrutiny, fVqfl^ jhe standpoints of economy, administrative coAveniencej 
implications fdM yalidity and efTective utilization of results, and impact 
on the economics, of test publishing. ' 

I should conimfent first that our experience is not in accord with Dr. 
Rankin's report that * 'teachers and administrators alike prefer tests 
which may be used with separate answer sheets." In the case of those 
tests which World Book Company makes availuble intoth versions— 
with and without separate answer sheets—there is no single instance 
in which the separate-answer-sheet edition is the more popular. In fact, 
there is fairly general resistance on the part of teachers and school ad- 
ministrators to the use of separate answer sheets at the lower grade 
levels (up to the fourth or fifth grad^ because ofrtlie materials-handling 
problems that these present to very young children. At.the uppe^ p'ade 
levels wf find, too* some lack of sympathy with use of separate answer 
sheets on the part of teachers w^o prefer to have tlie actual record of 

. . . , i^KS. » . ^- . . • * . _ '= - 
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the child-s worit before them rftther than jiMt an answer ilineU wliich m 
UUsuited to instruplional or diagnoitic application, 

The matters of relative economy, accuracy, and sfjcctti of ha|id vs. 
machine-scdrlng, and the eff^t on test^onstrucrf^iou and uiili^ailoii 
practices of use ^ separate answer sheets have becm ably diseussed 
elsewhereUnd I shall not comment on these issueB hqre other than to 
obaeryg thgfc 1^^ ^ft^^ fnr m& chipe-roo^ 

is not in every instance as ovei^owering as fir. f\ankin sqems to be 
saying. I muit not, however, neglect |to call your^ atteiiUon to certarn 
aconbmic proKlems implicit in the growing ule of separate answer slieets. 
It is obvious that use of a separate answer sheet, costing a fraction of 
what a test booklet costs, coupled, with repeated oise bft a test boQklel, 
means a considerably smaller return to tot publisher ^nd «uthor per 
studeat testad. You can readily imapne the dismay with whitih 1^ 
l^tened to Dr. Rankin's plea for test materials that will withstand \\\v.^ 
we#r arid tear of 'Hen or fifteen administrations." Jf this trend is to con- 
tinue, theft we must ask where the funds are tq come from for produu- 
tion of teats of the quaUty that school people are coming to demand. 

I susp^t that few test users realize the itiagnitude of the costs m- 
vohred in ieat development. The production of a major achievement 
battery, for example, calls^ for extended curricular research and text^ 
book analysis before a siii^e item is written; then for the writmg and 
experimental tryoutof periiaps as niany as 15,000 items, and for naUon^ 
wide standardization, not to mention the a^socinled^ research under- 
takings; such as investigatioiis'uf reliability, equivalence of fonps, and 
the like— calling for^an investment well into six (ignres. Ifow :mnny 
tests will have sufflcient sales to sustain such developmental coste^ if 
the per^examinee return is of the order represefited by the average price 
of an answer, sheet pr^twor plus a tenth ov fifteenth of the price of a 
booklet? 

We underslniid the Bchoolman's deBir|^>r economy, but we fuel that, 
it is necessary to view the matter of explw^t^« lor test matcnuls iii 
proper perspective. As nearly as our publishingiiWsi^^rigures ruvcal. 
the average-expenditure per olemenlary and secondarrs«h^L^P'l f*"" 
testing materialB is in the neighborhood of.20 cpnts^-certainly not more 
thW'SS cents. Imaiinel We spend 1300, 1400, even 1600 a year to 
educate a child and then worry about saving a few conts on teat malon* 
ata. What other expenditure yields commonsurate benefits to teaeher, 
dounselor, administrator and ptipil? I say to you, not entirely face- 
tiously, that what is good for the test-publishing industry is good for 
the test user. Exaggerated,^triving for economy in test materials can 
only dry up the sourcos of now and belter inslruincnts. 



96 



^3 



• ROGEB T. lENNON ? , 

' := . ■. ; ; ■ . ■ V". . ; . . , 

4 CQmpQmbkJJhits 

Thf^se€6nd probteia raised by Dr^jRankln is Uti need for units liiat 
are comparabte from field grade to grade. He urges at- 
tantioa to flndifig the unit t^i nvts'best the varied Fequirements of 
a good u^t in T^ hich to expreMeit results.**' We can readily agree that 
the vanety of vmys in which test scores are interpreted is sometiines 

, alnlitratbirr^nd^vP^^OT 
the dl^ire for simplification and uniformity, Howeverj th*appropriate» 
of a paFticiilar unit in which to expre^ scores depends on (1) the 
nature of the AliBty or function being measured; and (2) the use to^he 
made of the sco 'es* For certain functions, ^pressibn of scores in terms 
of age or grade icales is natural and meaningful; for others^ such units 
would have litth if ^ any value. Percehtiles, standard scores of whatever^, 
variety^ or any other type of unit of which I am awariei possess proper^ 
' ties^hich render 4hem suitable for certain types of testSj unsuited for 
others* The abiliies we measure are so, divefsej and the uses we make 
of our tests are k> varied, that I regard the universal, all-purpose unit* 
aa will-of-the-mip, pursuit of which is foredoomed to failure. 

At -equate and Eepreseniative NormMve D ' , 

call 



-ci ted 



Dr. Rankin 
bas^ OQ' adeqiia^ 
lations. I am Bun 
for aU purposes J 
this respect, aiid 
tests with quite 

Dr. Rankiii 
tween results of 
sumably attrib^ 
parable. This la 
unintended cons 
system of test de 
at difTerent time 
for hoping, that 
improvelfnents in 
body roT inforfna 
related to test p* 
precise definitior 
major test pubUj 
aegis of the C 
Institutie, which 
of testSj to be in 



ed, entirely properly, for provision of normative data 
e and representative samples of vaHous typefrof popu- 
that there-is no lest fopwhic^ nbrmative.data adequatq 
re available, but manifestly' progress has been made^ 
certainly there are now avail Ale. tests and batteries of 
respectal^le bodies pf normative data, ^ ^ ' 

the vexing problem of systenuitiu diiruruticuB be- 
tests purpurtifig to uieasure tliejome fuuctioiu pre= 
able to the fact that their norrri%roups are not.com- 
admittedly an unfortuiiate state of afrairs. It is an 
jquence of our independentv ^^rnetimes competitive, 
velopmeiit, and. of the facMfiatAtests are sta'ndardized 
and on difTererit pbpul^tfoijs. There are some grounds 
this situatipn ^11 jare^iinprovud, Firsti I would ctte 
nOrming technology; there iSj for example, a growing 
ion on cornmunity and ^eteol system characterisUcs 
rformance that will help us in the selection and more 
of norm groups. Secondly, and more speciOcaliyl the 
hers arei\ow engaged in a joint enterprise, under tl^ 
onilmittee ^BTests of the American pTextbook Publishers 
ooks to ^fc development of an "anchor testj" or series 
luded ir^ne standHrdiz&tion programs for all our^ re- 
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ipectivfe tisWr md to serve as i coi^on defining instrunient for their 
vaqouB norm ^i^* • j ^ 

MeanwbQe, mmy of the pjuMihers do' irovid the com^ 

pirabilit^ of results of Qprtain^pflheir tests, As for example,. School and 
College Ability Test v^Anri6^an €buneil Psychological old and n^w 
editions of California Achievernent Teit, and Metropolitan Achievement 
Te0i v^. Stanford Achievem ent Tetf. It is^ell to mention, however, 
that^**complarattlity** or couvertibility^ is not an absolute property of 
teat scores; scores which are found to be copiparable, or equivaient, in 
'dfte population will not necessarily be so in another— and hence the 
need, ideally, for local determinations of comparability. And I need / 
;^ not remind you that differences in norm groups are not the onty sources 
' of Rtek of comparabiUty "between tests. . ^ - 

/ DesirabililyMj %Tany More Tests 

Dr. Rafikin calls for production of aclnevement tests for every nwjor 
\' course, especially at the secfitfidftry levels and for lusts or groups of tests 
J **to measure more comprehensiveiy the achievement of pupils toward 
alf the outcpmes teachers seek," The question of whatT^ests will be 
developed and published is largely an economic one— alniDst entirely 
BO as far as the commercial test publishers are coneeriuU, and not 
neghglbly so even for our "non-profit*' associates. You may sure that 
test publishers will be vying with one another to produce tests in any 
fleld in which the demand is of a magnitude sufRcient to make the 
venture commercially worthwhile. Indeed^ it is true of test publishing 
as a whole that it goes well hleyond the publication of items that copz 
servdtive business practice mtght suggest. 1 wonder how many ,edu- 
Aoators realize that the very exmtence of the Icst-publrshingMndustry 
depends upon the sa^s of not more than thirty ielts or series^of tests 
and that these tests, irf efTQct, carry Iho hundreds of other test pub- 
'Plications? . ^ > 
' As rapidly - as school people demonstrate tliat they wi^J buy tests in 
given areas, tliose tests will^ get published. If, hbwevdr, author arid; 
publisher must contemplate on the one Ijand substantial investments 
of time and maney to dSvelop tests to mec^^ef more ex^ctiiig stanierdsi 
^ and on the other hand rewafls that are ifninimized through use of sepa- 
, rate'answerisheets, gelding a meager ^turn per-exQimnie^ you must ; 
not be.su^riped if many desirable tests kre left*un^uthored and un=; 
published* I* ? ^ M % ; 

I trust. Dr. Rankin, tfiat my re^ioii^to youy cqrtimentsjiave not 
appeared excessively i^#itive and that I have not appeared to dismiss 
lightly Jmy Of your pr^)8ali. We publishers know that our rfohtirtulng 

, - : \ 



iiid^p^- depeiras meeting iatiBfactoiily the lieeds of you'end^your 
feU6%:.^pgators, and believe me when I soy that we listen \vlth utnibst 

* attenUon to aU your recommendations. It wai Winilon Churchill who 
temarked, **I have'derived cQntinued benefit froni criticism at all periods 
of my life, and I do not*remeniber any tihie when I was ever short ^f 
it" Dr. Rankin^ I know that your remarks to us test makers have betfii 
intended in.no pritical sense; but I am certain they have been of benefit 

"•^^*tb^!ss, and thaf thg'^erieft uucrue to LliB Bohouf^dmlhii^ 

trators whose te8t=usihg problQms you presented so w?lL ' 




.The Guidance directors Problems 
and Suggestions for the Test Specialist 



.1 ^ • ^ ■ : /_ 

Perhaps the best way to answer the question implicit in the title of 
this talk is first to state the problems which confmtit the guidaiice|; 
director as he goes about his job on a day-to-day or year-to-year basis^ 
Then we can a^ ourselves the question as to whether the solution of 
these problems can be helped througli the use of teMs. If the answer is 
*'yei/' as I believe it is in much of our^work, then we have to ask our- 
selves in what ways present tests are inadequate and what can or ought 
to be done to reduce this inadequacy. 

Let us take a look then at some of the common tasks or problems 
confronting the guidance director, (Obviously not every guidance di- 
, r^tor will be taMd with all of these problems. Some of the problems 
may be the responsibility of other personnel in a particular school 
system.) We can begin at the beginning with a policy for school entrance 
age to kindergarten or grade one. Should all children be adraitted on 
^11 rigid chronological age basis or should .we try to take account of indi- 
vidual differences and have a flexible entrance ago policy? Let us assume 
that, for a variety of reasons, we adopt a flexMe school entrance age 
policy. We are still left with the question m to how we can determine 
whether a given child is ready to enter BchooK Teeter may be ready but 
not Charlie. How can we tell? 

Obviously we cannot tell by simply looking at the jhild or even by 
obtaining a ftiirly complete history on him.* We need some measures 
which will enable us to discover strengths and weaknesses in the child's 
ability to learn, in his learning achievement to date, in his desire to learn 
and in his social maturity. In other words we lieed measuring instruments 
which will enable us to engage in a diagnostic appraisal of this child as 
a candidate for school entrance and thualje able to predict whether or 
ndt he is good risk for admission. - 

Me there any measures available to do this job? At present I know 
of no iingle measure qv test which will provide'an answer to this problem. 
Nor do I know of any combination of measurea.the results of which can 
be compared objectively with a later criterion of success. We dojiave 
measures which can be administered by a cHnical psychologist. These 
can be combinetf in a cHnical Jhdgment with other data to indicate 
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itrairgths and weakriiMes and to make a^prediction as to p5t|ntial 
^cceas. But this is Very- expensive propef a and-as a practical matter 
is out of the question for most school syitems. Can the test experts 
develop Bonje reliable and valid instruments which do jiot need clinical 
training to admttuster and rnterpret'and which will be useful in. this ' 
pnAlm i ? V ■ k A L a:^^ — — - ' ---1 1;.. 

Let us move along in the educational stream whose headwaters we ^ ^ 
Have just ^ken a peep at. Here is Susafl in Grade III who is not doing 
as well as tha rest of her classmatei. A reading test indicates tli^t she - 
is reading on a low sedbnd grade level and the teacher's judgment agrees 
with this estimateJ^ Mother' has become very anxious' about Suian*s ^ 
learning and^aSi quite rightly,^ brought considerable pressure to bear 
upon the school to do something about the problem. Why is Susan not 
learning well? Here again we bavb a problem in dia^ 
there Are gaps in previous learning which are now handiGapping her and 
jvhich can be corrected by some good remedial teaching? Fortunately in 
the field of reading there are tests which do provide us with substantial 
clues as to lacks in skills and knowledge.. Unfortunately this is not 
tme of other areas of learning, except possibly for some of the simpler 
skills in arithmetic. 

Let MS assume that the particular gaps in Susan*s learning are dis- 
covered and remedial teaching is tried lo overcome them. Let us further , 
assume that Susan continuQp to have difriculty. Jlere we have a problem/ 
of attempting to assess Susan's capacity for learning. Here tests are v 
ijseful if they can tell us that Susan is or is not capable of learning any 
better than she is. We also need to know this as specifically as possible, 
h Susan's difncully with learning to read the result of a language dis- 
ability which interferes with "whole word" learning, thus making it 
necessary to u^ a detailed and rigorous phonetic approach? Or is too 
much anxiety/the disabling factor? We need to know also whether 
Susan is willing to learn or, to put it differontly, has resistances which 
interfere with the desired learning, , . : 

Tim type of problem applies all through the grades. More and more 
in our schools the counselor is expected to help pupils who have learning . 
difflculties. Very often we are not sure as to whether the learning dif- 
ficulty is due to lack of motivation or resistance to learning or whether 
there are some specific difficulties in learning which act as blocks to 
further learning* Tutorial or remedial techniques would apply in one 
instance and psychological counseling in another or both may be needed, 
I would guess that currently there is a terrific watte of remedial or / 
counseling time which at best is being uspd to find out that an alternative 
twhnique should be employed. Under ideal conditions at present this 
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is'bein^'aoni by .cUnioally traiued psycliQlogilli: but otviousbf. this ii 
'vei7 &£pei}aiv:e aildP imposiible for^ ; 

•Again can instruments fae dov#ped which doMiot iieed the, cUhioally 
' trained pe*n to administer i^nd which can do the twin jobs of diagnosis, 
an*.prediotionP I hasten to add that oven the clinically trained psy- 
"ISSirtfiMhisaifflcM 
' • reliable and' valid iriatrunients. , 

" Sb Mw,e have pffered examples of iuilialen^ 

iferiing difflculty'Let us proceed 4ii our journey to grade eight. Thi? is 
■ ' an important place to pause and consider one very crucial stage in edu- , 
' pation. It is usually ill grade ^ght that pupils (and their patents) are , 
- •■" ' asKed to-^ake i very important decision about a curriculum choice for 
■:].■> grade nine. Here again we are concerned with diagiipsis and prediction. 
■ ) V We need-to know what would be the bpsl Lvurriculum for tlA pupil to 
pursue in the immediate future^srades nine through tw#e. VVe^need 
% also to have some indication of goals tojtrivc for beyond high school. 
■ " Good counseling needs to take4iito nccomit both the innne|mte and 
ultimate goals to strive toward-'Choine-inaking whicli proccod^ pnlj on 
the theory of. year-to-year ^ipision making, with next yeai's choices^ 
. ' ' determined by tWs year's perforn.ancc, can bo a very wasle|ul process. 
" ' We are somewhat handicapped hero as yet- by nn inodequatlj theory ot 
. I .vocational development=allhough I hasten to add that nmth mterost^ 
" ing and useful work has be6n done recently hi this field by Roe, Super, 

' and Tiedeman, ' 
' ' Recognizing the lack'as yet of a fully kccptnlJle theoi-y of vocationa^ 
^ ' ■ development.'it seems to me, af a practitionef, that in order to lielp'a 
grade eight pupil make his decisipii we ought to hove rohablo and valid 
' ■ measuros of his achievement, nptiludes, iiituresls, .and mttoraticvi ior 
learninrwhat we havq to oifcr. Hero, too, the problem of .cpffiporabihty 
enters. It is not enough to have a batlory of tests, measuring different 
; ■ traits, standardized in the same population. Thfe is, of ^QUrse, superior 

to single testa standardized on difroront populations. But if two batteries 
published' By dilfe'rent companies und purporting to uieasurc sub-* 
. stantially the samq traits give (hfrcronl resiillfi, what should the practic- 
ing counselor think?' Is it sufflfciontly'clear that the so-callod^raits are 
artificial labels applied to mathematical artifocts? Which battery us 
' . really measuring the traits named? Both may have high coefTiciejits ol 
internal consistency. , 
> What do we have in the way of achiovemon t tests for our eighth grade 

boy which will be helpful to us for purposes of diagnosis and prediction? 
1/ we want to estimate the real qnality of this boy's abiUty in writing, 
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^ have to fell hack li^ly on teacher judgment. Some bright pupils : 
■with hi^h yp^ usage scores do riot seein to ' < V-y'i 

4 be^ble-to express ideas cleari Is this a matter of etjiotional baking? ' ; ' ■ " 

: Is It; lack of e with sini liar alDility and schopHrig - y 

[ lio v^yij^, w^t Which s ocial studies fests really tes t developed \ 

^gdwW in hiitbricaf tests of devclpped^BiJity TtF art 

and; mu6ic? V^at of tests of developed mechanical ability? ■ 

; Th6 idevMppment of multi^factor tepts 6f aptitud?^ a hopeful step _ \ 
f tpWards into for appraising aptitudes. But much obviously 

remaiiSs to fceMdne as Dr. Super's reviews of existing multr-factor tests * , , 

' in Vblidfhe 3^3Cy of the Pef^sonneland Guidance Journal tlmvly indicate. ^ . , 
v^Hfre^tht chief ic^j^ IB chiefly w^^ recognize thaVth^r/ ,- ^ 

^ burden of pr^bf camiot rest w holly with t^^^ ' 

V It is unrefisbnablmto expect that tests caii be developed in all^relis . , 
which will hayb universal yalidity for fell pqssiblq situations* TheN^st v , 

£V user must have sufficient sophistication to realize that;a given test even ' 
with k low validity coeflicient may be very valuable for some kinds of 

i use/ Also the test user must share the responsibility for determining 
validity for the particular use to wlrich lie wishes to put tlie test. Perhaps 
here the test specialist can help the practitioner devise relatively simjple' V : 
methods to he^ determino*Vaiidit>y Some of this has bciurdone by sonse ; 
publishers. . ^ 

AH of us rocognize the .hradequdc^^r interest tests for the eighth 
grade pupil Perhaps it is following a wilUof-thMvisp io expect that we 
can ever develop yalid interest tests for this ajc groupi I have a hunch \ ? 
that wheii we have a nialure theory of vocational dcvolppment thp.t * • . 

will not find this task as impossible as it seems now. At any rate I 
present it as a problem to ihc test specialist, > ' 

And the most difficult problem of all has to do with this eighth grade 
^oy*s motivation for learning. Can test^ be developed which measure ^ 

f motivation for different kinck of learningP Can the ''staying power'' of 
a pupil be measuredP If a bright boy as not loarning well in grade eight, 
is 'this a temporary matter? Or is his blocking suniciontly serious so that 
there is grave doubt as to whether any real desiiie to do systematic book 
learning can ever be developed? . ' ' ^ 

We have much advice on all sides today jabout the responsibility of 
guidance personnel to encourage intellectually able pupils to go on with 

* further education. Thinking only of the, individual pupil, and not of 
society*s needs* how do we know we really ought to do this with a par- 
ticular pupil? Aside from the very tricky problem of freedom of choice ^ . . - 
Ntliich is a real concern for^ counselors, how do we kiio\^ this bright pupil ^ 
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• has ot can have developed 
learriing? Perhaps this is 
but I present it to y^u as 
: Eoamininy further ti/f p\ 
inaking, it is appaferit tha 

/ as he tries to help a pupil t ^ 
or in. a particular jda. Cpll^ 
size, location, nature of studen 
standardi. #%tiilies of college 
to establlsh^^nilles of occupation! 
accuracy cQulck-tl/en probably be 
Paie (1) ajid others have already d( 
is a problem which, parlicular, 
school whose graduart^g^^ J^o st^ 
ing increasingly a prdblem.'Df tlie^i 




a real d^ire tor the pursjiH of 
:ondHh| technology of testing - 
it perplexing ones, 
national and vocational choice- 
BOMrttrlielFiliT^QptiB^elo 
for success in a giyeri college 
isifled in a variety *of ways:. 
, and estimate\)f acadeinic" 
jlished (similar to the eflWrt ; 
liction with a fair degree/ of 
[tJ understands that C.^, 
;^6me mjrk in Uiis direction, iflns 
^fronts the lar^" suburban high 
1^11 over the country . It is becoiii- 
ndent school as welh . 
lal process* which could be used 




There are other st^cs In tjie eduv^^^."- 
to illustrate problems in IteationalaM 

let US examine gtill an6tfierlind of i)rftiblcTn which confronts counselprs.^ 
Here is a 16-year old girl who tm^st been released^froin a reform saKooI 
and placed on^^olerShe may m& to work or return to high scho^: She ^ 
thooses to return;ip ichool. Shq^d we mlmit li^r? How stable is she uo^ 
t IlSr dflerifc Mr ccnnmitirieni hfid/bfien that of stabbing a boy. Will sbfe ^ 




be dangerous tc\ our otli^r pupHsP Here, loo^ i^ an extreme^ difBoMll 
problem.. ' ' ' ^ f . - \v " ■ ^ , - / '\ ' 

What ofJiic le^yeeU'^old mi \s\\L \^ becoming unma!Uj|eabl^ it 
beUer to let him If^ave sclitiolil Wmt can t^ing do y lieliM^/wit 
thi I problem? ' , / / ' 

Up to this point I Imv^ presented a few uxaniipi liMrfighting the 
major probl^^confrotiting guidaiire personnel todayi^hi^se have to 
do with Matters of p^lonalit^ dimculties. learning difli^lties, and edu- 
cationaUvocational choiee-niaking, J am not itnplyi|ig Uiese are 
tliree^parate tad discrete ai^as but Imve given examples which focussed 
in one area or th^ other for ^urposepOof ^Hlustration. ^ 

It is apparent by now that I conilderjadequate diagiiosis and predic- 
tion necessary in tlie work of the i^unselor and that tests can be useful 
tools for these purposes. The guidance offlcer. therefbre. needs to kngw 
how to select and use tests. The^iost usefid inlbrinatiori (n helplhg hini 
select and use Jests is that concertied with reliability, vulidily, and 
comparability /Let us examine eacltf of these areali frbni the viewpoint 

practicing guidance people. What I have to say flow about them in 
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il experience,* 



. ^-j: MM in part op t^e r^aulti^^f a quesOooiiiire which 1 sent to a grou^ 
^ of guidance directors and 

j ■■■ ? , , / ^ifwbiHiy dnd Comparab^^^ * v ' 
^ //t, M of US; WQttld a^jMia^ ft tM^^ 
/■ Wheq we sa^ thii wedy^ ahoadd s^^ th^kind qf reliabil- 

v ity wemeauan4rte axttjnt^ and we must tajLC into afccount the particular 
// punp^e fW- ^Wi^s^e are^using the test. Related to this problem of 
; re^i^ty, a& tests are used in sclwols, is the :jlrobIem ot'cbmparMUfy: 
; . Reliability and comparability; -'are parUdularly crucial for Icits of 
mental ability fnjelligenqe, or saholastic aptitude.^ Try as we will to 
/; idwelpp, restraint ahpu^^thfe yalue of a score obtained ft^rh: a group, test, 
\ ^ top often: teapHprs, j^rtiinistratorp, and parents (if they can flud out 
what the resultfis) t^eat thfe score of an ii#lligence test as an immutable 
assessmentharided down. fropa on high. i * 

To illustrate these (problems of reliability and coiiparability allow 
^ me to give you a somewhat extreni^ example but one that is not rare by 
any means, the lather of a tWrd gi^ade boy m another town came m 
oflQce late in the spring to say^hat he was.moving into our city that 
' summer and would our offlce test his boy for acad^ie aptitude as he 
felt the boy s ability 'had been misjudged: 0*e of our psychologists (a 
trained clinical psycliologist) administered a BinefVand obtained an 
r;Q/ofM2 which she said she felt was minimal and that if she had really 
wanted to ^ush the .boy .he probably would have gone higher. A group 
/ test was administered to this boy the next fall and an LQ. of 402 was 
obtained. In the sixth grade the rtext higher level of the same teit was 
administered and an LiQ. of 128 was obtained. Jn grade 8 a dijGferent 
group test was admimstered and an LQ, of 139 resulted. 'In grade 9 
another form of this same test was pven andJhe LQ, score was 130. 
Fortunately W/C give group tests of intelligence at periodic intervals and 
also occasionally administer individuaUests when we foel it necessary. 
If the individual test mud the 8th grade lest had never been given we 
would still have had I.Q/s of 102 for grade 4 128 for grade 6,'and 130 
for grade 9. Obyiously there was something completely out pt line on 

•The quesUonnaire wai sent to the memberH of -the Kirnatfir Boston 'Guidance 
Uub (cotiBistini for the trioBt pni t of guidance dlMc^m in East-Central Masaa- 
cliiiietU),^and a few guidance directors and counBelors about the country whoni I 
teoK'^^^^f ^rfi""™ J^ff rcc(ii%^d from 32 guidance workcrH in" East^Central ^ 
^ISn^S' "^-^ ''^ 100.000. U from cities of 

^#,000^00,000 and 16 from cities or towns under 25.000, H^he socio-economic atruc- 
turjg of t^ese 32 communities vary ^cmsiderably as do theljackgrounds in eacperienc^ ' 
apd trainiDf of the reipondent^, Heturng were oL^ined from 3 large ci tiei. 3 medium- 
J »«^S»tl^8^™ Ijmail city elsewhere aBout the coimtry. In addiUon, 1 state dirtctor ' 
of vocational guidBnce and 2 .county directors dhswcrcd and returned the mies^ 
tionnaires* ^ h"'s 

105 ,» > 
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1957 ImilaHom Gonferemm Tesling Pr^IMs, 

" the arado 4 score ancW^ rcasonnbly cautbu^unselot migJitW. 
oul^alrobable score of 130. I psperff thisAoys true potcntu^ 
S toll: What does this meaiffn terms of expectation, pred.ct.oR. 

-^n^'S^uSl^e iisi^tiar^feyrhave-higi^effl^^^^ 

of^«l I'Se!^, Acc-ding to the pubishers tl*y 
Wghly witf one anothe^Onemfght arp^ that this ^"^^^^ , 
and tLaififeWncei were caused by instab.hty w.th.n b«y To the best 
of my knowledge, however, taking into account ^11 we know about h m, 
. tWs^yXldbeiutodasanorraal.welUadj^^ 

can^helst expert provide co^nplors for problems of ^ 

of mlkbiUty^hd comparability. If wc give a tlnrd grade clnld an I.Q_ 
^^wS to be to that we willgot the BQmll.qMn grade six. or, if 
^IS.ne,:t.,|^h.ditoene^ ^}^-f 



/ 



a diffefent one, tliKMh^clUierence/.s u uut u............. 

tha ekhthkadc boy will be in tlie anrno position on an I.Q. 
::^^aUve tcfS. Sb wl^n he reaches the tw«mh grad. How.v^. , 
' S^i^t^ng JvLwide dcvi^lionstort^ , 
• SSrp-^herl iims to high reliability which are mostly based on, 

he li^e of internal consistency. Thi^ may actual y^b^ . 
CSjmmrabiHty rather than reliability since tl^ ^ . 

SerTtho^consti^cted by the same- author mu#of ^^Amly hav. 
Sin&ent than the grade three one in ordpr to test a higher luvel^^ 

SrX.FoUow,up retc^Ung with ^^f f « ^til^^EJt™! f 

deviations f c* many individuals, as does i^lustmg with th. sam^ mstm » 

. (cS^^ ^ stLbiUty)? This is, of COU.C, a 

1pm A true t(»St-retest "r" would bq based on givmg exactly the same 

uni r exSly the same conditions and to'exacUy the same |upds. , ^ 
Sum^ a^eglUlo amount^ of learnh|g obtained -^^^^ ^ 
the^t.lr a large amount of forgeltipg of the fP^fi^'^'^paS ^ 
, . ^«-n^avoMange^in,e„ 

v^f timp T would suffgest liiat ine tbbt icLtsv^ J *m . t « . 

^S:si^chp^orhomeasu.pow|na^i^n^ 
tr^its-raore specifically tests wlnclr are ^^clled f * 

temeence 'or scholastic , aptitude and o lier Bo^callcd. aptitude tests 
SeJe^p^ssageoftL^lnbtsoimpJ^^^^^^ 
/ i^imila oueht not to be seriously anectcd, ^ _ ^5 , 

^ ^ iSml thU is 'largely a problem 6f providing the oonsumerr .^th 
^ ^ " n^^i^tioA so !haA they could choos* and nte tests moreintak ■ 
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, Ugently. Do reliability cofefficients*vaw from. o^e sample to another and, 
if iOp how do the samples differ? Is tne standardizatiori|group UJtely to 
be more hiterQgeneous than the group whic|i tlie guidance director is 
tegting j Mpw jyut this hifl uen^ geliabi li ty P How does teacher admi n is- 
' tration affect reliability Were original "r's"'' obtained with classroom 
^ieachers as test administraJSfff? If uotp should this be clearly |tatedand 
the need for traiped administrators stressed if the' users wish to obtain 
=rdiability comparable to that obtained by th^ publisher? The infeeasing 
tenrf^ndy to provide 'instructions for recprduig results in terms of 
. range, *within which a given score might fluctuate, is all to. the good and" 
ihcjlild be acpelerated. This, will certainly holp in irlbr^ intelligent use 
of the test results. ' I * 

In my judgme^j this^niattert of relinbillty and comparability ne6ds 
much ^further tittention/ Much of the published data on rdliability co- 
efficients create too bptiftiistio an iitippcssibn. And a tesfc^ publkh'er 
obviously has no responsibility for reporting tlie comparaHilily^ ofjiis 
test and that of a rival publishegf Tha result is Uiat the consumer may 



particular 



unwittingly be the victii^ not o/ly of the error variance .of i 

but also of that variahce jfeie to the^ lack of comparablity of two 
esti of tlie sarfie niefrtjal fuiiction . ' \ ^ * 

In the ar'^a^i^ validity there seems to ijc nencl for clearer and^better 
statements J c^i^iri^ to more iiUelfigei^t use of testa by the consupier. Too 
V often statements on validity aruibo geuerar and seemingly remote from *t 
V the day-to-daV business of the school or. are too ^^Ai Ui the nature of a ^ 
. saltfe pitch. The type of vuliclity (coritciitj coitcurrent, predictive, or J 
construct) nceds^to be clarilied, , ^ ^ ' ^ - - 

We need to know under what^^onditfons and for what kind of saniple 
the yalidity data were obtained. For e:^niple, /s the test of Utile qr no 
^ use wi^iii. pupils who liave rnadin^^^fieicfticies?^^^ ^ ^ 

ivfbre deftiiled information is'^ieeded as to what criteria were ii|ed to 
determine the validity of achievc^^^^it tosts. Which textfor curricuhuns 
arere used? Tlie guidance director wants to^know whether they ar^tne 
p mum as ot similar to th^si^ii Tils school ^stom, lest we fall m^the ^ 
- traft of tests de|erniiniiig curriculum rather than the reverse,) # 

The problem of overlapping levels of'achicvenicnt test" batteries is one 
^jlhat needs examining. This one probably includes reliability, compara- 
jbility^ validity and theHgood sense of tlie teachers ao'd counselors, tJn- ^ 
^foftuijately the habit ImM 
of gr^de placement. Thisnabit has been" reinforcfed by iSmfe parents 
(encouraged by some of^the hysterical literature attacking public ichool 
i ' ^ ^ I . - V , ^ ■ 
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feedu^tion) aemanding'^to. Know exaict. grade accomplishment of their 
chUdren. Jh* sixth grade teacher thus eagerly leizes' upon ta^hieve- 
l ment.teitTjattery which the pfcUsherB^^l^^^^ . , .' 

gradeplam^TOti f ^herptipfli-Sh^^ ^ 

■ ■ is designed for frades S and 6. She isAreriipleased'because many ofi her • ^ 

- children get seventh, eippthrsnd ninth grade V ; 

. ^ ' thisfacthappUy to«hfparentsof Ihose^^^^^^ • ^ 

' entrfjumof high school MchereJnOUobeMrt^^ ' - 

' higher form (lev«l C) of the same achievement battery is adminij(tered, ^ 
What happeni? Eqiiivaiint grade,scores six nfontha to a year less than; 
* I those'obtained in gracfe 6"are the result. If^e sdiool sys^m is lucky it ' - 

can confine the resulting dispute to the principa^^^ 
' • . " junior high schools. If the parents get ,wi^d of^^l^ difference then their • ' 
' suspicions about soft education in the felementar/ sc^^ ' 
V iBitthpresponiibiUty oftheteBtusersmoUto usescqi^^ ^ 
I- are open t6 all kinds of misinterEretation, or is it that of the specialist . 
* |0 suggertonly those mettiods of scoring which are the least susceptible , 

' ' to piJsinterpretation? , %i ' ui - 

It would be helpful t" have md^e precise stattmiei^^^^^ .* 

^ F/or example: "Thilk tiest which'wi "have called mechaniml aplilude does 

' ^ noti^ itself predict marks in high sohobl industrial arts coursep-sp well : ' . 

-.aa'atest of VerM aptitude, \ml it can be useA to inrprove predictionB * 

over those based on verbal aptitude tei scores -nloiie." This^fijcami^ " ^ - 

illustrates the need for emphasizing caution iij^ at-CT;pting titles bf testi, 

andnames of "factors'^and in the use of'profile sheets for makiiig com- ^ 

' parisons. Item analysis •studies ^ one grMp'of pupils may bo ui^valid ^ 

• for another due to Ibcal' phraseology and tllrtis diBtorl test result^. .Iof 

■ example, in one reading >est the phrnse "falling fast" occurs a n.d the 

flralple choice gives rain and, slorm omong other words. Ram-^the ^ . 

— , cVect resBpnse bul'in somelocalities f*w of us think of ram. as ftpng . 

^ "fast"— rain faUs "hard*' A storm' mlglrt fnll fast. As a result although ■, 

it is one of the earlier andfeasier quosliohs, in one loc ality it was missed, 

by SO' per pent of the pupils. * t » » 

Our previous discussions on eduOTtional-vocational clTOce-raa^^^^ • 

ilhistrate the need for continuing long-range validity studie^ wid to , 
lelntp test'results more and more ckj.sely to luch matters as sudffcsB on ^ 
. the-job 'and -in college. As I have \ii^loated before, however, tfie.test 
user must share some of the burden of deteTniining validity 



. ' ' ' ' Adminislraliiv. Problems 
AUow me to, pause a moment hqre to comment on adimnislrative 
aspects of tesflrij. By this I mean givrng 'teats, scoring them, racording 



ERIC 



' - ! = ^ EDWARD LANDY 

resulti, costs, and such matters as format and printing. I am doing thig 
'iinaply^b^ause these itenas hava been given a great deal of attention 
^oth by practitisners and test specialists. This is not to disparage their 
importance but meg^^j^^j^^ 

represent the most ift^Qltar^ or primary issues which face the guidance' 
' director in the use of tests.;All test experts are aware oLthese problems 
and much attention has been given to tlieui. The lessen the tegts do 
^what are claimed for them the more important tliese secondary problems I 
beapme. I have a huiich that if tests were constructed which really 
provided answers to fundamental questions, guidance directors, coun- 
itiors. teachefs, and principals might be willing to expend a good deal 
9f time and energy trfget thgsb nnsweis, It i^l^vhen the tests are pf' 
minor yaJue or when there is considerable uncertainty 'tis to what the = 
test scores really do-mean, that the tests Tt^d better be cheap, easy to 
admimster.^core, aud r^ord if anyone m to use tliein, I j^nsten M/add 
that if we can get tests which really providg^ anWers^nd are also easy 
to score, «fc,, then life would be wonderfuL I3ut, speaking very much k 
an airfateur to experts, I have a sneaking no! ion that there may be an 
insoluble problem here. It just may not be possible to get the right 
kinds of answers the ea^ way. Abd perliaps we hdd better stress much 
more the problem of getting^he right kinds of aHs\V^rl and worry a bit 
less about ease of administratiou, > 
V \ \ JJ] ' 

The preceding discussioiHPon reliability, comparability, and validity 
stressed the'needHir more intelligent use of lusts^ The booklets T^c/imca/ 
Becopimmdalions for PsyclwlogiM Tests and Diagnostw Techniques (2) 
and Xechnical Recommendai ions for Achierementyrests (3) sponsoi^ed in 
coiHmon by 'A.P.A., A.E.R,A,, and N,C.M,u!r are very helpful for ' 
this purpose. Some of the respoiffeibihty for bringing this about, how^ * 
ever, is directly that^of the test specialist in producing better toslsl more 
clearly defined, and v^th Biorp explicit instruutions as to their limita- 
tion^as well as uses, IIow caii this be brought ahQut? 
^ ' If there were ro^need for publishers 4o concentrate on tests salable 
to large market^, there wQuld be much benefit in focusing on specific, 
welUdefined populations in specific and welUdefined situatbnsrif a test 
did not ^ve lo fit potentially every pupil, even in selected schools, but ^ 
could belimited in Jts applicability to pupils with specified character^ 
istics, it would gain considerably in usefulness. At present, too many 
school people, expecting too much of tests that have been ''oversold,"' ' 
may feel diiilluiioned about the .usefulness bf testing !n generaL The 
test .pubUiherswh^ claim directly or by implication, more than a test in 
^ many instances cai^ deliver, are doing themselvea a long-range dfs- 

. ^ m * 




Imilalwml Conference on Testing Problems . ' ^ 

iervico. That tlio test caii deliver for "lome of the customers does' not. 
orevent alienation of others.- , ' , , . ' ' i 

Subject matter specialists are coftsujted to help linprove content 

volidity oPt""*^ not 'committees of guidance .people bo u sed 

to try to help determinp the kinds of prodiotions that^aro most UBcful 
for them? The test pubUshers could then, direct tliou- research to llu> 
contribution of test data to such predicliDns. , ' 

Fdr etanlple, we need very badly to have measures of molivation for 
specific areas pf learning. All counBclors have had c^nenoe with the 
% Wght pupil/who is not doing well in a parifculnr subject. I recall 6ne 
very bright boy, in particular, with whom I counseled a few years ago. 
He was failing in English. It was obvious ho had an extensive vocabulary 
and he spoke in carlfully slruclured sbntcnces such that one could hear . ^ 
■ the punctuation marki -dropping into plaoc, 11 Q perfornipd very well on 
objective, standardize^ts in Bnglisii but was failifig large y because 
he would rtot hand in an Acomposi lions. Ho protested to me that he was 
interested in writing, bu^ just somehow couldn't bring Inmself lo sit 
down and write. His father i| & succossful free-lance writer and -this 
' blocking may have been the result of involved relationships with limi. 
Was there any value in attempting to work with this boy on a supportive . 
level encouraging him- to write? Was this a problem of a profound re- 
sistance which cbuld be. uiiravoled. if at all, only by psychothempyP 
Or wai there simply no inturest in writing which wa^ a function ol tins 
boy's normal devollt'"cntal growth and not iho by-product oi sonic 

pathologj? . . . . ,. , , 1 i j . * 

Counselors are concorncd also uboul .jblaimng roUablo aiid valid tcst^ 
to -ifieasm-e personality trails such as coopomlion, initmlivc, responsi^ 
' bility They would like to have n'^tost of values" 'suitable for secondiiry 
school pupils. They want to know if we can roo^ly measure study ^nbits 

and attitudes, i,- ■ 

These are samples of questions niid pioblcnis. wlncli a consull.i g 
.coniniittoe of guidance peoplft might pose to the lest spuciahst lo help 
the test specialist better nuffil the ncecM-bf guidance personnel. , > 
One other a^ropirifls ihv the separate publisi.jfll houses^ lo havo 



wne oincr amprouiiii lo lu- i>>v. , = 

available field cofv^ultaiits who know mensurcnienl and have had eon- 

siderable practical e.\peiieiu;^«. , „ t ^r / 

Another solution m\$il bu lotorni an indcpendeiil Bureau of Meamre^ 
menh Used in 'Sihnols to act in an advisory and selecliVc capacity lor 
the public schools. Such a Bufoan CouW also help .in educating parents 
and the teneral public as to the function and limilntions of tests in an 
educatioWl program. It could engage in^tudics pf cowparahilily, not 
for the kke of finding fault, but for providing uselul data for the coun-^ 

, i 10 " ^ 

Jut '. . „ 
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selor. The Bureau could searcli for existing pfoblertis^id necKls, ptMiiailB 
♦ thifough the device ^the Advisory Committer suggested above, and 
^ act as a clearing house for relaying tliis irifurmaUoH^to tent' specialists 
« and publishers. • ' . ' . - ' ■ ■ - . . ? 

^ It ismy convict ion that are fo lb wing a wIFUof-thlswisp^if, as so 
many practitioners seorii to feel, desirably, we thiirf^ wccaii rn6y4,towards 
greater simplicity in this matter of testing,^ T lie very uses ^br which 
guidarti^ people want tests are cqmplex and ^becpniffng njore complex, 
■ , To provide tlie kinds of tests wh'rcli will really help a particular guidance 
person in a particular community do liis ^ob well will require an in- 
' creasingly complex technplogyr If tli6 l^al guidance people are to Ha 
the cletenmners of which tcBts tliey should iK§e, then tlie already wide 
gap whicli exiits be^vedii the test specialisb and the test user will con- 
tinue to widen* As the leclmolp^^y increaHes in eoniplexity or perhaps 
even changes in some Inipurtant I'undameulals, it wilbbecome intJeas- 
jngly dinicnlt for the test user to qlioose wisely and to u^e welL 

Is the. answer t^be fouiid In local school systems hiring specfahsts^ 
in measurement? If so, will they detei rnine whicli tests are to be jused 
for guidarice purposes or s^ill they act in a (*onsulUng eapacily to the 
guidance director P Can in-service Imining provide the answerP This is 
'certainly necessary in one form or another, if teachers and counselors 
are*to use tests intelligently in their respective rol|*s, If we are to^have 
» any in-service trairiing wliioli will liave ariy real impact on tlie^teucliers 
uncPcounA^lors we luul better do sornelhiiig about devising aii appropri^ 
^ate training program. Mo§t of our training programs in measurement 
are devised to produce the expert in nieaHureni(Mit.|We need a consumer's J 
training programiithat is something else thnri a watered down version 
of tlie program fot^ the prospective cx^Hm t. \ 

If we really could develop a knovvIe|lgeaTjle aiuLsophisHcated group 
of userB, the questions posed woiikl tend evetilually to be solved. Mean- 
while there rests a large etliica! responsibility^ upon the test specialists 
to make sni^e that the relatively unsophisticated test users art;* helped 
" to blroose tests wisely 11 n(ll() use jiU'*i^ 

lUci^, n, \\, "Thv r<)|l('||f* (Mis iiinnnu^rit nrul lU n^IatioiiHliip to ihn jirrdietion tjf 
cujUt f?o H\i<'('i*Hs/' I*iipt*r Tviul Ut tlio Ann rirnn pHyriiolijpU'nl AHHotijation lun- 
vcyilitiii, Nrw Vorli, HopUMnhnr 1957,, ^ ' > 

Tt-cljnical He€(Hnmvn(hii(yris for i-sychuhifiirnl T rsf^s at id Diuqnoslw Techniques, 
(l?r^>part*d by a joint ccmunitU't^ of Uio A,P,A\ A K^H^A.,, and N,C.M.U.E,) The " 
Aiiu^ririin, PHych(>It>gicoI AHHiiciution, WVX^ Sixt^^uutli Htretd,^ N.W,, WnHhington 
h;bx:.\ . ' ■ ^ . ^ ; ' 

Tech ri icul Hcco m menduiiiiti s for A ch ie vvnivnt Tenh. ( I * ro p < i rt* c I h y t?u ^ ( -o in 1 1 1 1 1 1 eda 
on^TvBt Stondaretg of A.H.H.A. nml N.C.M.lMC. Hnd HidorH;*(l by» the ^.A>) 
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A generation ago, A.^. Otis acliievcd inimorlality^)y iiiventing iho 
self-administering lest; you all know, l;am sure, lliat bit of rneasure- 
raent folklore wliicK has it that, aiix:rous to avoid the time-consun^ing 
work of individual test admihistration, he devised the first gronp test 
with self-administerimg directr»ns. Test technology then introduced a ^ 
* second labor-saving device-, the self-suoring test, and tlic names of Clapp 
and Young became household, or rtfther, schpolhouse 'words u^itir ma=' 
chines virtually oilplaced the '*self with machine scoring. Recognizing 
. the"iieed» for Jocal Mrnjs, test pubKshers ncxl. made available special 
^ forms tp assist test/conm*ners Tn developing their- own norms, and the 
^ self-norming test f;ame into beiiig.^ Oddly enough, no test-!distributor 
with merchandiBing propensities has attempted to capitalize on that 
terni aiid no test constructor has achieved inmiortality bj associating , 
his name .with a so-called self=norniing test. And now, Dr. Lakdy points 
out. a fourth road to fame, fortune, aiidHht^.future awaits the test con- 
structor who first devises and markets an instrument wliich will solve 
the problem of specific validity, that is, the first self-validating tesL 
'Surlra test would meet the consumer's need for data on the predictive 
' value\<)f tests for courses iii school, fbi^colleges to which his studeiits. 
\ go, foV jobs in companies which hire his students. Dr. Landy has recog- 
nized that publishers canuot accuinulate and make these local validity 
data available tu ^very school whft ii needs them— it is diflicult enough 
to validate for more general types of predict ion^ The implication: con- 
suiners need also to be producers, not of tests, but of test data. Of which 
inyre fff^ny^osing paragraphs, 

■ Tlw Need for Diagnostic Tests 

Morte regularly than the topic of validity, the need for diagnostic 
tests coincs up time and again in Dr. Landy's talk. But tlie diagnostic 
tests which our speaker asks for are nut ijie types of diagiiostic tests 
with which we are familiar, they are sonicthing jnuch more complex. 
^We are. used to thinking of diagngstie tests of two types: those which 
assess strengths and weaknesses in a skill subject such as reading or 
arithmetic, and those wliich diagnose the mode of persoiial adjustment. 
In the former type of test we seek to determine, for example, the arith- 
metic processes which a pupil has failed to master, so that he may be^ 
given the understanding of proci*ss or drilled in tlie use of the process 
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whicH^has given him trouble. In the latter type, we analyze modes of 
reaction in order to get an understapding of behavior dynamics, so as 
to use this to guide counseUng, psychotherapy, or placement. 

But we have heard a request for tests whiffi^ will diagnose more 
complex forms of behavfor, tests which will tell a counselor whether 
aptitude, skill, of emotion is the reason for poor performance; The re- 
quest stems from a better understanding of Jehavior on the part of 
guidance workers, from a recognition of the factihat poor achievement 
may be due to Jack of skill in the use of a Jool, to lack of aptitude for 
learning to us^ the tooL ot to emotion which inhibits tlie student from 
learning to use or from making use of the tool 
; Typically, of course, these complex diagnostic and treatnfent decisions 
^ have been made by the use of a number of different devices which in= 
elude individual intelligence tests, achievement tests, observations of the 
pupil at work, personality inventories, projective techniques, aifd inter- 
views. The use of such methods, as Dr, Landy points out, calls for 
considerable training on^ the part of the user and considerable time in 
■ which to use them with the student. Current trends in_ the training of 
school, psychologists, upgrading them from mere givers of individual 
intelligence tests or processors of group test data to clinicians trained 
at the doctoral level, reflect the recognition of this fact, evenViougli no 
comparable upgrading is evident among school counselors. But ac- 
companying this recogijition in the case of school psychologists is the 
awareness of the fac^ that the number of pupils is Increasffig more 
rapidly than the number of school psychologists, that time for more 
. intensive wgrk with more pupils simply is not available. Teachers need 
treatment-suggestions more often and more rapidly than school psy- 
chologists can make them. 

The cry for a new type of diagnostic test, which will identify the 
relative rotes of aptitude, skill, and emotion, arises from the fact that 
mass production machines are often developed as substitute^ for man- 
power, to do the missing craftsman's job. Counselors are insufliciently 
trained in psychology to do a clinician's work and they have workloads 
which prohibit doing such work even whe4 as in some notable instances, 
tJiey acquire the knowledge and skill But ihey do have the psychologicai 
sophistication to recognize that behavior, even in doing arithmetic or 
in reading, is complex, and they want tools which will enable them to 
give teachers the help that teachers want and need. 

It probably seems unreasonable to expect a test, or even a test bat^tery, 
to maJce a clinical diagnosis. The task seems much too complex fcff a 
device or a gadget, appears to call for a human diagnostician. But there 
was a time when appraising intelligence, or judging maatcry of English, 
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or assessing personality, seemed to be the kind of conjplex clinical task 
whicli only human judgment could perform; there was a time when 
predicting perfbrmance seemed also to be something for men, not 1^^^ 
machines: to do. But we are all familiar with the fact that tests gencral4»g«. 
make these evaluations of intelligence, achievement, and pcrsonnhty 
better than individual judges. We all know that, as Meehl s review 
showed they predict success better tlian judges. Perhaps it is not such 
a wild idea, after all, to ask for a test, u battery, or a formula wliich 
will do a better or-a faster job of diagnosing the causes of poorperiorm= 
ance tlmn can be done by a psychologist, 

Tlie Need for Teams of Psychologists and Teiihniqans 
I'ntil someone invents a truly diagnostic test battery and formula, 
such as Dr Landy has asked for, we must rely on people. But properly 
trained people, lie has pointed out, are too costly to do llic job un the 
scale on which it must be done. The American I'BychologicalAssociatioii 
nrquires 'about 1,500 new^meinbers each your, but only a few ol them 
are fully trained school, clinical, or counseling psyclu.logisls. and the 
recently retooled universily Irnining programs will not produce a greatly 
increased number. School counselors are ihererore asked to do the job 
with the currently available tests. With only one yar ol training in 
guidance, and generally not more than two courses in measurement, they 
have generally had the good sense to recognize their inndequac.es as 

diagnosticians. , . i i • 

What is the answer when the fools of mass production are lackiiig. 
when master craftsmen are too few. and journey men do not have the 
necessary 4kills-> It consists, uf course, nl^ building teams ol masters and 
journeymen, in which the journeymen enable the luaHler craltsiiian to 
multiply Ins services by workhig with and through them 1 nj qualified 
psychologist can do less testing, less interviewing, hiniscll. He can rely 
more on testing done by persons with less extensive and intensive trnuH 
ing than his. supervised by him anajspeciali.ing in H'c use of certain 
lypcs of tools. One does not have to be « Ph. D. in order to be able to 
give a Biiiet^a technician enn do that. The real skill is in the mterpre= 
lation A team of psycho=lechnicians working with one psychologist 
could do more work, and do it better, than the same number ol conii= 
selors and psychologists working Mingly. 

It is true that, in the last 15 years, clinical psychology has been 
unwilling to consider this t^pe of p.ofessional slruclure, it has insisted 
on the doctorate or nothing. The speciallics of school psychology and 
counseUng psychology, on the other hand, have thought ni terms o 
training at botb the doctoral and sub^doctoral levels. But Ihc^liave not 
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applied the team concept, long used and abused by rnedicine, to their 
own field. Since abuse of an idea does not necessarily invalidate itj liie 
team concept might well be tried In diagnosis in the educationnl setting. 

' . Olher IS ew Tests ^ 

Dn Landy has asked for some otlu^r new tcBts, particularly tests of 
valuiss usable at the higirschool level and tests of niolivatioiL It would 
be intereBting'to develop this topic, but time dor^s uul pennit. Instead, 
I nrnst move on to, more debatable issues. ^ 

A Ted Consumers' Union 

Another need raised by Dr. Landy is that lor help in evaluating the 
large number of available tests arid selecting the most appropriate (or 
a particular use, in pooling information about testa, and i<i educating 
the public coiieerning the use of tests. This need is real to me, thanks 
to a committee on tlie^ American Psychological Association which play cd 
with the idea of a Bureau of Test Staiidards while working an a code of 
ethics, aiid thereby raised a storm of protest. Work on the ethics of 
testing led to the establishment of another committee, th(> Ainctioii of 
which was to develop technical standards for psychological tests. Both 
committees backed away from the idea of any ofllcial (^valuation of 
tests; instead, they prelerred to (clarify Itandards and stress the de- 
velopment of informed mor^. But Dr. Landy's HUggestion points up 
another possibility: that of a Test Consumers' rnion, comparable to 
sunilar organizations whicli evaluate arid report on various types of 
conimodities for their mend)ers, Huros' Yfarhook now does something 
of this sort, the recent series e^f . articles on nudtilactor tests in the 
Personnel and Guidanve Journal docs it in a dillerc^ut way and on a 
smaller scale, and some test publishers ofier consulting services in which 
public service and inerchinidisirig motives are vaiiousty mixed. But 
these are individual or corporative eflorls. Perhaps then? is room for a 
test coiisumers' union inatle up ^of school syst(^ms organi/ed to pool 
experience, to share normative and validity data, to protect themselves, 
Snd to educate the public;. Perhaps we should paraphrase the ancient 
warning and say, Camint mvrvatnrvs! 

The TraininfjKoJ Test I sers 

I return, in closing, to the training of test nser^ Dr. Laiidy has raised 
this question very clearly in his statement that most training in testing 
has been aimed at test specialists. But there are, as I see it, two kinds 
of test specialists: the psychologists of clinical and counseling, persuasions 
who develop a high degree of diagnostic skill in using tests with indi- 



115 



1957 Imilalional Conference on Testing Problems 

viduals, and the psychologists of a melrio persuaiion who construct the 
lem Dr Landy lumps both types of psychqlogists togother^ as test ^ 
specialists in contrasting them with School counselors. 

However, it seems to me that what we have heard is actually a pica 
for the training of more school counselors as-tcst specal.sts, not 6s 
psychomelrioians, true, but as diaenosticians, in the manner charac- 
S^f cUnical and counseling psychologists. But with the shortages 
S now exist in thoseHelds it does not seem lil^ly that tl^c nu^^ 
of highly qualified test users, trained to work independently, is hkely 
to be Neatly increased. I am therefore inclined to re^nterpre Dr. 
Landy's plea as an indication that we need to find bettor ways of tmm- 
ing somf psychometrists or psychological examiners at the masters 
level able to work as members of psycho^diagnostic teams led by fully 
oualified school, clinical, or counseling psycholofists, and otlier psy, 
chomet'rists trained as collectors and analyzers of normative and vahd.ty 
data as members of test development teams led by test constructors. 

It is interesting "that, in this 'conference on testing, m this talk o . 
what the guidance director wants from test specialists is not so much 
b^ter instruments that are asked for, althougli we have heard such 
requests, as better users of instruments which are now ava.lab e, better 
ways of making available instruments yield tlje information which tliey 
L be made to contribute. This is not the first time^the teaching of 
Sing has come up at these conlerences-tlie 1953 Conferee made 
the subject a major topic on the program. But an exanunatioii of the 
Proceedings confirms Dr. Landy 's Point : the focus was on testing as 
mea«nt, not on testing as underslanding people. To many psy= 
dSsts^i h a doctorate these may seem to be one and the same 
Sg But to the user of tests in schools they ^re not. The ques ion 
e4re is, could wc meet the needs of more people, by t-ching t^ing 
as. the understanding of people rather than as the understanding of 
measurement? 



I- 





Participants— 195T Invitational 

Abraham, A, A,, Florida A & M UiiU 

verpity , ' 

AFFi,ERBAcai, Janet, ProCe^lunal ExaniU 

hotion Service, New York City "i 
Ah^ia^n, J, Stanjeyf Cornull Universily 
ALBniGHTs Frank West. Orange (N.j.) 

PjibHc Schools . 
AtMANi Juhn E,, Bg.^ton Uaiverai'ty 
f^PCASTAsii Anne, Ford ham Usiv(|rBity 
ANDEnsbN, Edward L,, Educaiioual 
' Testing Service. 

Anderson, Harry l£Tt^Jr*i Fort' Bliss, 

Texai^ . ^ - ^ 

' Anderson, Pauline K,, Nmv York Statt^ 

Department of I^bor 
Anderson, Scarvia B.|y|^Jucatiunal TeHt- 

■ing SiM vice 
Angqf^, AVijIiam 11. , Educatioiial Tent- 

ing Servicti 
AitMSTRONn, Fred (i., tj, H. Steel Voim- 

dation ( ^ 

.Ahonow, Miriam S., Now York City 

^r^mrd tjf Ednoution 
AaHKNiAN.^Seth, Spr ingfield (iM^hh.) ^ 
" ('pllfgt' . , ^ " ' 

Atkins* W iUiani 11., HnlgiTH UriiverHity 
BAifrn, DiHialtJ E.. neruTal Eiectrid 

('nmpany 

FiANNON, ( '.Iiarles J., (Irosby Htgh Sghoul. 

NNaUnlHiry, Crinn, 
lUnnAcfe* Ilerhert^D^ New Ycirk^Stale 

Departnieiit af Civil Service 
lUnnY, RoberrF., HocheHter (N.Y.) ? 

Board of Education 
Hartnik, Rubert V,, Educational Test- 

ing Service 
Bates, Margaret A,, Educational Test- 

ing Service 
Beck, Hubert 'Park, The City College 

tif New York 
Bedard, Joseph A., New Britain (Conn,) 

I^nblic St'hooiR » ' 
BEMENTi Dorothy M,, Northampton 

(Mass,) School for Girls 
BpNNETT, George K", The Psyehologicai 

Corporation 



ndix 

Conference on Testing Pro^ems » 

Fennbtt, Mrs, George K,, The Psycho- 
logical Corporation *^ 

BENNETf, Balph, New York City * 

FJenson, Arthur E., Educational Testing 
Service . ' 

B^TSON, T. G,, We*5tern Electric Co,, 
Inc. 

Berbie, Ralph F., University of Minne- 
sota 

Berg, Joel, King Philip Schonl, West 

tfertford, ,Coun. 
BERGEn, Bernard, New York, City De- 
partment or Personnel 
Bergesen, B. E., Personnel PresH, h\c 

Princeton, N, J, - , 

Berne, Ellis, U. ^. Department of 

IJeaHh, Education and Welfare 
Bicknell, John, New York State Do- 

j^artrnenl of Education 
BiRCH,'Dorothy L., Educational Testing 

Serviqe % 
BixLER, Harold IL, Western Carolina^ 

CoUege,, Cuilowhee, N, C. 
BiJGH, Harold, World Book Company 
B|jTz, Sidney, New Y^ork State Credit 

Unron Ix^ogue 
Bi.ommerm. Paul,' State UniverHity of 

l(?wa 

Bloom, Benjiuuln, liniveiHity of Chicago 
B<JLAi^o, Buth F., Clamhridge ^Mans.) 
^ Bofiid of Education ^ 
l^oiJ.ENBACiiEn, Jonn, (Tineinrmti (Ohio) 

Public Sch(i6lM 
Braca, Husan E,, Oc*?aiiHlde High School, 

Fprest Hills, New York 
Brandt, Hyman, Araerican Occupational 

Therapy Association , New York City 
Bransfohd, Thomas L., New York State 

Department of Civil Service 
BnAUN, Ixjuis !L, Bronxviile (N,Y,) 

Public Schools 
Bhetnai-l, Willluni B,, Educational Test- 

ing* Service 
Bridges, Claude F., Washington, D,C, 
bpiERLY, Justin W\, Denver (Colo,) 

Public Schools 



f 



117 



1957 Imiialioriql CotiS^ence on Testing Problems 



Bnistow, William H,, Bureau of Cur- 
riculum nesearch, New York City 

BnoDEniCK,* J. Lawrence, Y.M.C.A,, 
New York City ^ ^ 

BnoDiNsKY, B. A, C. Croft Publi- : 
cations ' ^ 

BnoLYER, Cooil R., rsow YurH Stato r 
Department t)f Civil Service 

BnowN. Fred S., Great Neek (N-YO 
Pubiic Schoftls i 

Browt^j, Lcanna, EducatiQjiaV TeBting 
Service 

B^AN, Miriam M., Butgers tJiu\;erHity 
BuYAN, J. ^i^d\ National Education As= 

Boelation ^ 
BucHnEiMBn. Aruold, New Sui^ ('ity 

Board kf Higher pdueuLluu^ ^, 
Burdock, L, BloniotricH nesearch. 

New York City 
Burke, Paul J , Boll Telephone Laho- 

ratorics. 

BunNiiAM, Paul S., Yale UnivcMHlty ^ 
Bunos, OBCar K., lliitgerH UuiverHity 
Euros, -Mrs; Oscar K:, Gryphon l-roHs, 

Highland Park, N^ J. 
BuTCiiBR, Herbert B., New Jeft^ey Du- 
^ partmeqrft of Civil Service ^ 
Cadwell, Dorothy !L B.. Civil Sorvtee 

CoinmlHsiou of Carindu 
Calvo, Georgia, Edurnticsrial TcHtinj^ 
* Service i 

CAPrH,'Marlnn P.^ Vi^iniy H^tah? College 
Ckiu^ON, J, Hpeneer, liiiivrMKil^ of 

OregciU \ 1 

Carlhon, K,ato, Henuitlial I^dueatton 

Ceiiter* Wn^liingtoii* C. 
Caruolu Jtjhi! P., IlnrvHrd t^ilverHity 
. CAiipATER, KugfiiM D.i Purcuu ot Navul 
lYrHonnel 

^"^CahiiMAN, Jerunii: P.. Archdiuct HUii Vo- 
cational St^viee, New York City 

C^YNi^. BeniArd B., Ginn oiid Company 

C^APPKU., Barllett IC. H., New Yink 
Militai;^ Aeadi^^iy f 

CilAUNClcf, Ilehry, Kdueationul Testing 
Service ^ ^ 

Chovnowski, M., lyucati/Mird 'resting 
Service 

. Christensen' Cliirord. New York State 
Department, of Education 



CHuncHiLU Buth .D,, Antioch College 

CiARK Pamela, Educational Testing 

_ - ■ ' I 

Service 

CiEARY, Robert, Edueational Testing 

~ " ■ ■ ■ 4" .■ 
Servlce 

Clemans, William V., National Board of 
Medical Exaniiners, Philadelphia 

CL[5NDENEN,Dorothy, The Psychological 
Cor:poratiun_ 

Cores, Jeanne IVk, Echieati/mul Testing 
Service , f'' 

boFFMAN^iUiam E., Educational Test- 

'\ vug 8i^^3l(fe — / 

eouRN; Iniilip S., Montrflir (N J,) Stat^ 

I Teaeht'rs College 1 ^ 
Co|.E, JoseplP|^^, Uni^|ersity of 

»i}-^.Jt.?iiester " , 
am^^M. Elizabetha^, New York City 
¥(jeotieinai Educuti<^i and Extension 
Bojird 

CoLLiNH, Goler E.,>^ciuth CaroUna State 

Ctjllege ^ 
CoNuAD, Uerhert B.. S. Onice of Etlu^ 

catWHi ^ 
^Conway, B., Department of Educa= 

ticar, Vietoria, B. C Canada 
CuvuLXNii, Uvnmn A., Pennsylvaiiia . 

State Civil Service Connnission 
CoiifsEliLHl^N, J. Ih. Mahh^t Associates, 

New York City ^ ' 

(Unn , Chnrles, Philadelphia Department 

of Personnel 
CnANE, Ihu'old L,, Jr., Ktliie^uionul Test- 
ing Service 
CuANK, I^^rcy F., University of Maine 
CuAVKN. ICthePCase, P(iiyMmic Instil 

tiiti' of nrotjklyti * 
CHAWFom), J. IL, University of Maine 
Ciu?NH()N, Wilhehnina M = , I'ahner Me- 
niorinl Itistitute, Sedalia, North Caro= , 
Una 

(AiMMimi^. Mary B., ntistou Public 
Sclioois 

CuRirrtiN, Edward VI, Rnoxville, Tenn. 
CuHETijN, MrB. EdNAu'tl E., lUmxville, 

Tenii. . 
Ci;m\AN, AnnM., University (sf Con- 

neeticut 

CuRTi;sf, Wyltua B., Catholic University 
of 'America 



1 1« 



ERIC 



PARTIGIPANTS 



Cynamon, Manuel, Brookfyn CoHegu 
CzuKOR, John C, NuW York City Do- 
. partmcnt of PerBonner 
Daileyj John Bured^ of Naval Per- 
sQiinyl ' 

pALYi FranciB J, /New York State ."toc- 

partment of Education 
Damrin, Dora E., Educational TesUng 

Survice ' , 

Davidson, Helen J-L, The City College 

of New York < 
Dean, E. Douglas^ Educational TeBtiiig 
* Service 

' DENisoNj Violet, Educational TestJilg 
Service * . 

DEnwiNj Edward D., Crosby Higl^^ 
Schoulj Wat^uburyj Conneclirnit' 

De Sales, Sister NJ[., rWary HiU College, 
BulTalo, New Yorkf 

Detciien, LUy, Chatham College 

DE Thomas, Janu, EducAlic^al Tostiiig 
^ Service - ^ 

Diamond, Lorrtirfne K., Teachers C{)llef'ts 
Colunibia Uiiiversfty 
^ Diamond, M. David, uA-erBlde lIoHpiUd, 
Bronx, New York 

DiEDEnicn, Paul B,, Educationnl Test- 
ing Service 

DiGGs, Franklin B., New York Cit^De- 
partment of Personnel 

Dion, Robert, California Te^t Burenu 

Dixon, Robert E., Oberlin Ctillege 

DonBiN, Jolm E., Educntiiinnl Testing' 
Service 

DoppELT, Jerome E,, The Psycholuglt-nl 

Corporation 
DoRDiCK, Mildred, Ediicatitinal Trsting 

Service 

Do^vNESi Margaret New Ycirk Stat* 

Departraent of Civil Sirvicc 
DnAoosiTz, Anna, EducaSionnl Testhig 

Service 

Drake, L. E., University of Wiseonsin 
DuBNiCK, Lester, Munfcipa! Collegea^of 

New York 
'DuKER, -Sam, Brooklyn College 
DuNTf, Frances E,, Brown Univernity 
Dyer, Henry S,, pducalional Testing 

Service -. 



' Ebel, Robert L., Eduoational Testing 
Servieu 

Eddy, Robert P., Rutgufa University 
Edel, Wilbur, New York City Depart^ 

ment of Pereonnel . 
Edelst^in, J. Dqvid, New York City 

St\idy of Mentnlly llandicapped 

Children - 
Edwardb, Wiriifred, Irvington (N. J.) 

High School 
Eells, Kennelli, University of UllriclN 
Elder, Samiiel^,, Mathematics Irreorgjo- 

rated, WaBhiiigton, D. C. 
ENGELnART, ' Max^ D., Chicago Cit^ 
t J*inior Collegy " / _ 

EftaELiiART, Mrs. Max D., Chicnfo, IIJ. 
Epstein, "Bertram, The Ci|y CoHcfc of 

New York 
Epstein, J^larinn 0., Educalicinal Test^ 

ing Service ^ 
FAULoa. iSrucei Edueational TY-^?ting, 

Service k * 

Fay, Pntd J., New York 8tat'e Depart- 

incnt cjf Civil Service . 
Feinihcrg, JVIortinier R., yiic njjseareh 

IjiBtUute of Afnerica, Ipc., Ne% Vbrk 
'City 

FeL-dt, Lt-onard S., UniverBitv 6f I()wa ^ 
I'ENDnici4, Paiil, WuHtern Klectric Cctin^ 

piiiiy, Inc. . / * . 

Fenoli.oha, (^(M)rge M., Ilau^hton ^ 

Mifllin Cuinpnny ? ' ' 

FKNHTEnMAcnEn, (iiiy stNF, Ediieajitnial 

Tcssting Servici! 
I'lninijHON, John P., The Pingry School, 

Elizabeth, N. J. " 
FEHGtiHON^ l^%inflrd W,, Life InHiirariee ' 

Ageiiey Mayagc^ineiit Associailori 
FiELDH, Niiia, OpEj^^ikin 'lloll Academy, 

Atlantic Uighlund?*, N. J. 
» FiFER, Gordon, Test Besedrch Service, 

Inc., New York City 
FiNpLEV, Warren G., Atlanta (Ga.) 

Board of EdneatitJii 
Fink, Duvid R., Univernity of Maine 
^FiNKLE, Robert B., Metropolitan Life 

* InRnranec Cnmpany 
Fisher, Evelyn M,, New LlncolriSohool, 

New York City 



4W 




^ ' * 1957 Imii^^l Conjerence on Tesiing Prd^ems 

Fmi^N. Joshua- A., CQllrge Entrance doLDSTEiN, l^ro a. cMoW VmjMvMs 

Exanimation Board J . ISjfcdical Cijlkge, ^ J/ 

• . Fi^NAGAN. J6hn C,*Anicrican In^^ (Uk^an, Hamuli ^*^w ^ rtrk St.ile' 

- - . , fcA- Research. Pit^burgh ^ Dfc^artment nj ^iduratlofU >^ . 

'f: rLESCH. Mari^H., E4ubay^nal|pPr ; sonnd HrBnarch Field Ac tivity, NAii 
V Service - / ' ' ■ t = . * . 

- FoLUN. KaAariBe. Reirfeaio! Educotbn . Ghay, Mrn- Lylo Bla%, (iilnfaiu^rhn^I^ 
' tester, Washington, D. C. ^ J Baltimore ' ^ 

FbRLANo Georgg/board of Educattcfn/ GiUBnca^N, Warren. Ilar^^d linivTply 
Brooklyn. N. Y, ^' "^nrrm.^^^^u^^^^ ^ 

\' ' FoRLA^or Mrs George. FluBhm^CN. Y.) JVlass. }^ ; ^ ' V^"^ ^ 

' ' Ai^ic ScHo<tlB^^ - ■ ^ UiUNSTED, AliJii 1%, Na^y beparlnuMit 

-FbRRELTER^ Gertrude. Webl Side High Ihrnss, Heuben llJif^ A ^tudy of thn 
Setebl, Newark.' J. ^ American liigh School; New;^ City 

' \ M4a||b; Thelma R., Cncm fUdse (N. J.) ^GuERiuEao, Muhael A. The ^^.^oU 
^ - HjgJj Sehoolf H . , ' lege uf New \ ork / / 

IfRKA^/Hof ardi., Jr., Ecluca^iunal 1 est^ , ( Uaj.iK^a^Nr Harul^ .LducdticmHij 1 ent^ 

ing E^rv^ ' ^ mg^^^rvie^ \J 

FREDERl^Wr^orman, Edi^ationi^ GutiuuK, Gcu^rge Nt. PeiinHylvania btMq 

Te^ttrng Service, / ^ ' ^P,' . Univorsi^ ; , ^ „ 

Freemaih. Pmil ^1-. KducationalTeHtinR Haagen. U Ile.K. Ohio \\oHleyan I 



Service 



jerMjty 



Ke^NCll, Benjainfri J., New Y()i^ itate HAnKN, Eli^abHh. Teaeher^. C.nljeK^s 

Department nf Civil Service ^ " Oolumbia Um^orHity . 

French John W:, Kdricational Testing Hag!^, Bosa A.. Ir>^ngU)n (N. S.) Pubb<' ^ ^ 

.... ^. 



Service 

.FiUCKE, Bt'iiiio = , UnivrrHity of ^firhi- 

Friedman, Sidij^, Biyrau (»f Naval 

PerBoniiP^ \ 
FRL)TCllE¥>^re<f IV. F= Depajtnient 

cif Agrirulture ' ^ 

F'lJLTON, ReBeB.J.. nfew York <-itiy Hoard 

t)f Ediication 
(iAi.LACHER. Henriettn, iCdtu^llonal 

Testing Service ^^^S^ 
(lARDNKR, Eric F., Syrnriiffin fnivorKity 
(jmK, Hei^nAL, ARHoeiiit^nfi nf /VtntM'iein! 

Medical (lollegeH 
Georgia, Sistijr Nt., Ib^HaxyHill ( Uflb^ps 

Bufl'alo, N. Y. I 
(iKRBEUiCK, J. Hayinniid, liniverNity nf 

Connecticut . 
(iiDDiNGS. Frank. Springfield (MaHM ) 

Trade High School 
iEASBn, RoberC, University of F^itl^i^ 

^hiirgh 



Schooln 

IlAnMAN. Elmer firrenwirh (Clnnjij 

Piiblle SthofilH 
i I AiJ.,' Alfred, IbitgerH I'niverHity 
I Iai,!... Robert Ci.. \Tanter Ifall Srltnfjl. 

flanibritlge. MaRH 
i^RVEY. Philip ft, Kflucationai TeHtinjT 

Service f 
IlAHTiNf'H, J. rhnnuU, I nivtMHit^ <»f 
llltnoi?^ 

l||vHMAN,ill(>wHrd J , Nnliunal Seienee 

Foundatiun 
HaVO!-;n, ICri<S Public Sers if-e, Newark, 

N= J. 

IIku.v. KrfteHt k.^ Jr., Center Uiv f'Hy = 

i-buhjgit:al Service^ WaMiingtcm, D 
IIkatun. Kefineth L. lleaton. Ployd and 

Wutsnn. Philadelphia 
IIkii., r^miH M.. Brooklyn Colkge 
lluth B., Glendale, Ohio 
Helmick, John S . Educational Teiting 
S^rviGe 



i lnmmMM, Fred I , Ednrational Testing ITEMPlirL!. John Edneafionnn^^Mfip 



Spr% it'e 



ERIC 



\ PARTICTPA,IVTS ' 



HEnnlCK, JaijUH, lUigdi^ IkIiuhI C'oU KtnJJ^n, Fraiiklin J.,'C'(Hiinnniity 1 uh^n^ 



h'ge of liducfltinii 
,!lf;YMAN. MnrshaU N., KalU (lluncli 
Virgiiiia 



Sf'tirch, Nuticnfil SnlinlurHliip S<>i vir-c* 
iS, Fiuid fur N I ■J?rn StiidcnU * Xrw Yurk 
C'.itv 



LlifsiuiNY^iiH, A, N., Stat*' IhiivciKily nf KF/Lrl^v, \\. Vml. il S. NHViii Srlionl of 

^Xvlation .MiHiuMins V'^fimicDlai Mn. 
KKi^icii, Clit^ntU'' K., StUiulHrcl 

Kkhn. Qiiiialtl, lIiii^i'iHity i if IMil^r'p* n t 
kiiAN, lUili Z., iM'dt'ral PiiHlii! ScrNiiT 

f "nifSriiiHHinii, Ni'SV ^ <H'k (lily 
kiMnAi,!,, tCliHaiK'lli ( Kduratiniial 

KiNfi, Jijii^llian, l'U\* Vntul i'av lYiv \d- 

s aiiuffiH'Ml ijf Kdiiruthin 
kiNi^Hiu ny, Mtn. Marioii. lleiiii'diHl , 

P dsKUitii ill- < 'mti-r, ^^ n^^iiiij^tiiiii 
k iMKpATniCK, Kp^n'^t I ].. Jn'thaiiy 
(W. Vaj CiM^^ 



Iowa ' 
I liTCiftiocK, Aiihur A.. Alnt-rican^ Vvr- 

Nniijipl and (luidaiu'i'^ As.^cjcialinn ' 
llrmNOign,' W illiaiir Sfalt' r.oHi'gcs 
, I'tinnsyVvaiua 

HoLIJiH, Esther, Thv I*Hyj;Ih iIii^ic nl Cnr- 
puratiniiV ) s ■ 



1. I. 



II{}PMANN,H^iJj«Tt i*., BtJurd for llij^ln 
f'ldiicaluni, MisHotiil Synod of tin' 
I jilhtTaM' ( 'liurti! ^ 

iron.OWlT/, MiltoTl \\ ., QuLMMlS (. ^ciUfgi' 

I Iiii>!>LKsTc»N, Mdilli M = , Kducatioiiul 

llittiiiFH, J(ihii k.? fnt«'rnati!H!a| l^imrnr,^ 
Mnt'liiiirM f jfrptfrailun " 



hiANCi, Frndorit'k lA;, Kduoatuiiial 'IVst^ 

iiig St'rvict' 
koCH, John Jr.; Millhiirii J.) Jr. 

I f igh St'luMjl ^ ^^.^ 
'koMiYAMA, I'^ii'Iii. rrikyri I iiii s i 'r^i Ty of, 
iLtiuculiuii 



'^''^^^^^"^ ' ' ! KoHTUiK, Max M., SUitr Tvm^hvv^ (U-^ 



Ini':iii':tj', %1. Ik, Nal inriaP Src'iniJ^^^ 

Ag'Tiry y * ^ ' 
Jauk^oN," fenn^luH iVniiHylvniiia 

StaU' I 'iiivnrHily' ' 

1 li'lli HtdnMil 



li'^i- at HpHlnn 
Is. [?A TfnvsiHL, Da^id W.. fiij^'aii ^iah' 
. _ Uiiivrf^ity ' 
i\HAff*4H, Ih^'H, M^icyV^ Nt'sv Ynrk 
kniiiM, JcjHt'ph h\, Kordliairi HJtiisiMH'it y 

(jjlunihia UnisfrHil^ 
k V A«A(:fiiJN, \\, ( ^Uij-^tun UnivjM^^ity 



J\H!M^N, '^NuthuM. National I^ aK'ii' fnr LaDijkk, Ctiarh^M 1 1. S. AriniKl F<)rr*'?4 



!\iu?4ing, N«'W York ("ity 



^!!Klitutt^ Matl 



JuiiNj^N, A. IV!Mhprt(m,,Nownrk (N J.) * kAMKE, Tinu liisui Slatf 'l^rarfuMs 



Ctj^pg^ of Miigineoniip ■ .CloIIr-pr- 

Johnson/ Patricia, Juiucalioiial 'iWliug Landy, lidward, Nrwton (MaHK.) i'liblir 



S*_!rvit_u» 



Schn 



Juoi,a, Arvo K., Midiigan Stati^ Unl- iMNt;, Uerharl, Thv i lly Ccjllogt* of 

Nrsv Ytirk ^ , 
LANfiMriii, c:. M.. 'riu' P?4^<'Injhigit;al 
'nrpnrat inii 



K ABACK, (lijldm lUiUi, 1 ho ( -Ity ('nkli g* 
nf Nt^w York 

KAiii!iiKii5 K/ kiMUU'tli, DofmrtiriiJ'nl of Lanniiolm, ( i. \ I-itlta'atioiial IVstlii, 

tho. Army 
Karl, Mailelhif^ Hrouklyn, i\. \, ^ 
KKLKlinn, Cirrftory St. AiiHi*Iin's 

Cullogt!, iManc'hc^sti'r. N. II, 



I .AYTUN. \\ illjnr 1,., I 'iii vcraity of Miiiiit^' 
sola 

kid^cjii). \\ illiani k.. IMiithii- UiiivtirHity 



121 



^18 



i957 rnviialionat Conference ^h^esting Problems ^ 



LEBdw» Daniel B,{ Now York CHy^De- 
* partment of Persojinel 

Lee, ^x)uise F,, Remedial Education 
Center, W€ishijigton, D, C. 
/ 'LennoW,^ Roger T,, World Book Com- 

, -^V^M^T^F., Richard, Educational Tyslhig 

Lif^dui|Tj E. F,, State Umvursity^ of 

LiTTEkiCK,' W.' S,, The Harley Sc^iool 
^ Rodhljdr. N, Y 



Masia, Bertram fe.. Science Ucmearch. 
*' Associates 

MatiikUs, ClieMtl^ O., O^^Vesleyan 

Uiiiversity ^ ^ 

Maxwrl-l, Sheona, Ecliiiburgn, Sentland 
MAxwELLi Jamys, "reacliers Collogo, 

Columbia University 
McCann, Forbes E., Philadelphia Du- 

.partment of Personiiel * 
McConn, Richard B., Philadelphia Do- 
^ partm¥?nt of Pen^onnol 
^IcCnACKEN. Charles W., Trenton (N J,) 



LiTTYAFrank, New Jersay Sts^ DepartT^ * Statu Teachei s College ^ 

/'rnent of Civil Service ^ MclNTinE, Paul IL, University of Nnsf 

LoHMAN, Maurice A., New York State ^ IlnmpHhire 

* Department of Education McKenzie, Francis W., Darien (Conn.) 

LomftLLE, IE George. Dade*Cdunty 1^ ^ Btmrd of Education 

Board of Public Instruction, Miami, 4 MuLaughun, Kenneth F\, Florida State 

Fia. -I . University 



Long, Eqnis, The City College of New 
York ' ' ^ 

LoRDi Frederick^ Educatic5nal Testing 
Service^, 

Lord, ftifrloy IL* Educational Testing 
^ Service 

LonEE, Ray* Ixiuisiana State Uni- 
versity 

LoRGE, Irving, "Teachers Cttllcge, ilo- 
luinbia University 

LoHGE, Sarah W., Jarnes Miinrue High 
School, New York City 

LoBi, Carrie R., Board of Education, 
Newark, N. J, 

LouGiinAN, George A., St. Fraiieis Cul= 
lege, Brooklyn, N. Y. 

LiTHK, Louis T,, Nurwalki Conii, 

Lirrz, Orpha M., Montclair (N, ,b) Slate 
Teuchurs CcjUege 

Lyman, irnward B.j University of Cin- 
cinnati 

Lyonh, William A., New YoA State 

Depnrtmetit of Educalioii 
MacK AY, James L., South San Antotilf* 

(Texas) Schools 
M^noNV; Thomas E, New Jersi'y Hlale 

department of Civil Service * 
Manuel, HerBchel T., University of 

Texas 

1 Marston, Helen M., Educational Tt'st^ 
iiig Service 



M^Ianus, E'O F., Jr.,, University of 

Connecticut 
McC^JiTTv, John V,, University of 
Florida 

MEAnE^ Mfirtin J,. Fordham Uiiiversity 
Meulev, UiHiald NL, Muniripal CollcgCH 

of New Vork'C^ity 
Mki.ville, S. D., Educatsonal TeHUngj 
- Her vim \ 
MEnnv, l\(>bert W., Harvard UiiiversitV 
MEhwm. Jack C Syracuse University 
Memsick. Samutd, I^dneatioiial Testing 
SmI vice 

Mkta, EllitHt, Little Nerk, P^nw York 
MicuEi.!, (HMic, Metropolitan Life Ii!= 

surancc? C'cnnpaiiy 
MiiJiH. NIatthew B., Ihirace ManiH 

Linccihi LiNtitulis' Nt'W York City 
MiLntnj.ANO, John K., Uiiiversity uf 

Michlgnii 

MiiJ.Aiu), Keiiiietb Dt-partmeiit uf 

nt^fensc 

MiijjiH, JSjMii'ard S., A .Study of the 
Amerirun High School, New York 
City 

Miij.En, Ilaitild T,, ILju. 
Ciini|!ariy 

Miller, William N., U/S. Gbvernment 
Mitchell, Mrs. Blythe C World Book 
Cjunpany 



1 IVlMn ' 



In 



9 



EKLC 



D^partnunil thu 



Newton (fVIass.) 



^ S * RARTICIPANTS 

* 1 • . 
Mitchell, Col 

Army 
Mitchell, Robert H, 

High School 
MiTTMAN, Arthur^ State University of 
' Iowa 

MiTZBL, Harold E., The City Cbllege of 

New York 
MoLLENKOPP, \^Jiaita G.j Procter & 

Uaniblti ^ 
MooHE, James W,, CaHforiiia/ Statu 

Scholarship Commission 
MoHG^My, Henry H., Th^^gychologiuul 

Corporation 
Morrison, J, Caycej The Puerto Rican 

* Study ^ 

MosELY, rUis?^^WiHuonHiii State De- 
partment of xMucation 

MuiiiHEADj David B,, Michigan State 
University 

MuifinKAD, Peter P,, Now York State 
Department of Education 
, MuNGER, A. M Standard Oil Co. (N J.) 

Myer.s, Sheldofi S.j Educatiunal Tentirig 
Service 

Nalp.y, ThoniuM, Univur.sity i)f lUicKjt^ 
Inhuid 

Nedelsky, Leo, /Jniversity oT Cliicago 
Nevin, MargsBfet 
Service 



NoLL, Victor H., 
versity * ^ 



Educational Testing 
Michiga^ State Uni-^ 
Educational lUKMirds 
Hnading IiiHtitutc of 



North, Robert D 

Bureau 
N tm Lo w, Sa n u J n I , 

Nosow, Sigmund, Michigan Stiltu Urii- 
vcrsity 

Olsen, Marjorie, Edueatioiial ToMlng 
Service 

Orellana, Egidio, InBtitulu Podugogico, 

Santiago, Chile 
OaLKANai Joseph B., Georgd WaKliin^tt)n_ 

High Hehool, New York City » 
OscARSON, Donald, The Taft SJIkxjI, 

Watertown, Conn* 
OxTOBY, Toby, Technical Reportn, Inc., 

New York City 
Packard, Albert G.\ Department of Edu= 

catioDi Baltimore, Md. 



Palmer, Orville, Ediitational Tenting 
Service 

Palmer, Osmond E., Michigan State 

U!ii<^>r?^ity 
Partington, Dorothy B^, Educational 

Testing Service 
pABnALiANv Siruon, Comnuniity Service 

SocirtV,. New York City, 
Patton, James B., Jr., Virginia State 
f, Dupartnieu^ of Education - 



pEABonv/ l^lizabi'th R., Gr^ton ScIum)] 

Grcjtons Mnns, 
Perlman, Mihlied, New York City Dr= 

partnient of Personnel 
Pehloff, nobert, Seienee Restmrch An- 
Hnt'iutes 

Perry, Williuin D., UniverHity tiiyVorth 

Carolina 1 , 

I'eters, Prank H., Oliio State University 
Peterson, ('aria A., I^ducational Tunt- 

ing Servicif 
PtcTERHON, Donald A., Life Insurance 

Agency Managenu'iit ^AsHocintiiJU 
PiULP(>rrr*Euiily L., Queens College 
|-!EnH(iN, (leiirge A., QuvvtiH (Icjlh^ge 
PlTCn|:R, Barbara, iCducational Testiyi: 

Service ^ 
Pollack, Ncjrjuan (L, New York State 

Department of Civil Service 
Preston, Braxton, lulucationnl TeHting 
Sj:=Eviee 

Price, I.^mUj, Ifniondale High Selio!)!, 

Valley Slreoni, N. Y. 
QifLNN, Jolui H., Jr.i NS'cuid Book CMni= 

Harineau, I^juis, Pratt Institute, 
Bnioklyn 

Rarinowitz, Willinni, Now York City 

Board (if Iligher Education 
Uanoali>, IFarvey, New York State De= 

partinent of Civil StTvice 
Rankin, Paul T,, Detroit Public SehoolH 
Ransoiioff, PriHcilln B,, C^jusultaiit As- 

Bociates, Iiic, Long Brancli, N, ,L 
Rapparlie, John H,, OweriH-Rlinois 

GlaBH Cornpany ' 
Raskin, Evelyn, Brooklyn College 
Ray, William S., Pennsylvania State 

University 



123 



'^0 



1957 Invitational Conhrence on Testing Problems 



Read/ Thomas* Moumee Valloy (Ohici)' 
Country Day School 

Reed, Anna K,, New York SlBtr Do- 
partnieat of Civil ^*rvice 

Reiner, Harry, York City Depart- 
ment of Personhel 

Hem^ers, H, Hm Purdue iJniversity ' 

Heuter, William H,, Educational TcbC'- 
iiig Ser^k^lce ' . 

rA-h* Jcannf, 4Authefford (N.^J.) Ilh|h 
School 

RiCHAnDs, Roger Thb pB^chological 

- ■ 

Corporation * ^ 

Jr,, The PBycHoUigirnl 



Educational TrHtin^ 



Ricks, JanieB H. 

Corporation 
RiEDi^.^Norto t V 

Servirn * 
rUMAUn'En,' Jank K., iCdurational 1>Kt=" 

ing St^rvici^ 
RoRBiNSi Irving, Quet^fis ColU^go 
Robertson, Marguoritn If., University 

of Connecticut r 
HonLYEH, William A., The Chont^^ 

School Wallingford, Conn. 
RosiNsKi,' Edwin F., UmScrsity of 

BuITqIo ^ ' 

Worn, *Mm: C. A., The Pingry Hcliool, 

Elizabeth, N. J. 
nomi, 0. b., Schenuctady'CN. Y.) Puhlic 
School^ 

^ULON, Pi X, Harvard University 
RuNDQUiST, E. Am Ale|ondria, Va, 
RuNKEL, Philip J., University of Illint)!^^ 
Hait, ■ Edwardi Rensselaer Polytecluiit; 
InHtitute 

Salzbbhg, Florence, Queens Collogo 
Sami^r, Joseph, Veterans Adminmlta- 

tion, Washington, D. C. 
SANDEnH, Edward, UVjnuuia Ccillegr, 

Clarenioiit, Califs 
^^anFOhd, Nevitti Vas^sar College 
Saunders, Davj^f^., Edneational TeHt= 

ing S*jrvice 
Sawin, Enoch L, U. Air I^ijree I^OTC 

Headquartura, Montgomery, Ala. 
= Scates, Alice Y., U. Oniee of Fdu^ 

cation 

Sen A PI RO, Harold B., Young & Huliinnrn, 
Inc, ' 

SqiiEPERs, J. M., prineelon UnlverHity 



ScilRADBR, William B^. 

^'estiiig Servid^, 
f^niiULMA'N, Frances H., luBtitute f«n 

Career Guidaruc, New York City 
ScHULMAN, Jay, Institute for ^Career 
^ Guidance, New Yorfi City 
ScHULTz, pouglns, Pennsylvania iStole 

University 
ScinVAHTz, Alfred, University cif Dela- 
ware ' \ 
HciiwAiiT/,, ,loe, Mitehel Air Force Basc^ 
Scott,' C, Wlnfield, Rutgers Utiiversity 
Scott, 'Winifred Starhuck, New Bnnis- 

wick. N, J,, 
HEA^iionE, Itarolfl (i., The I'Hychologieal 

Corporatiorr 
Skibel,' Deari W., Hducalioiiul Tenting- 
Servicc- 

Set^Eh^ Charles J., New York City iJe- 

partment of Personnel 
Sforza, Richard F., New York State 

DeparLment of Civil Service 
SiiANNBR, William M., Cnlifornia Te-it 

Bureau 

SiiAPAN, Norma, Roardof ChiUl VN elfare, 

^rrenU>n, N.J. 
SnAfip, Catherine ih, KclnralionHl Test- 
ing Service 
RHAYCOpr, MuritJii F,, AnuMiean rnntu 

tnte for Research, WaHhington, C., 
Shea, Ulster T., Hauppauge School 

SysU-m, L, I., Y. 
SiiEAa, Bruce, New York Stale Depart^ 

nient rif Educatiim 
SfiiEi.Ds, Mary, National Ix^ague ftir 

Nursing, Inc. New York City 
SiiiMriERG, Benjnniin, ICducational Test- 

ing Service 
Simon, Dorothj M., New York VMy 

nepartinent of Personnel 
Simpson, Enznbeth A., Illinois Institnt«v 

of Technology 
SiTnaEAVEH, Hosedith, Teachers Colh^ge, 

Ccilnmhia University 
Smit, Jo Arnie. Arlington, Virginia 
Smith, Alexander F., New Haven 

(Conn.) State Teachers Coliege 
Smith, Allan University cjf Cnti- 

iinrtu'nl 



121 



PARTICIPANTS 



Smith, Aiiti Z., EdiicatinnQl TuHting 
Service 



STRONrf, Edward K., Jr., t^tHiifnrd Urii = 
ynrsity ^ 



Smith, Vmml D., Office-.of Naval lie- Htuart, Williarn, Hduralicmal Tcj^ting 



search 



Service 



Smith, Rnbert" E,, '^ducflUnnal Testing HTUi.nAUM,Jfar<)kl, IClnduirRt, New Yjjrk 



SuPEU, Donald I'l, .Teat'l^er?^ ^I'ollege, 
erHity 



feervioe • 

Smith, William Reed, IJxii^«rsity of Utah " Columbia UuiverHity 

B N oDG R A ss , Robert, Ed uc a 1 1 o n a 1 TeB t= ^ S % k ndk e N , ■ J t > h n n , Tea f 1 h C o 1 1 e ge , 

ing Service : ' Norw'ay ^ 

SoHMEn , Kejiiie th D . , M i II bu rn ( N , J . ) S w a n^on , Va\ wa r d ( K , Ui i i v e rs i t y uf 



High School 
Solomon^ Rob*»rt, Eduoaticmor Testing 

Service' ^ » 

SoiJTHEH, Mar^ Taylor, TiswerMlUi 

HchooL Wilmington, Deb 
Spaney, Emma, QueoiiH Ccj^lege -f 
SpAi/LUINO, (Jeraldine, Durham, Ncjrlh 

Carolina ' 
SpEER, GeoFgy_ S., liliiiniB InHtitulft of 

Tec'hnology ^ « 
SbitNCEni Richard E., Jacksnti (Nfich.) 

Public Schools 
Spefvcer, W. Douglaa, Nt>w Yoili City 
Spencer, \VeBley G., ('am bridge, iNlaR'?. 
Stahi.* Sijzannp, Educationa! Testing 

Service ■ ^ , 
Stake, n<ibert. Edueatiunal Testliig 

Servit'e 

SrALLMAN, Frederick B., Nesv Vijrk 

Telephone Company 
Statleiu CharleH R., Univer^^ity uf 

(_^hicag^o 

SIecklein, Jnhii E., Tliiiversjty iif 



^^i!l!!e^ioU^ 

Swift. Evuretl L.^ The Peddi(^ St huol, 

llightsttf|ti!, J. 
Sw iNEFOHir, !■ raii('*»?4,' iMhu atioriar,^! esi = 

inp Service ' ^ 
SvFEnD, Jailet,, Ediicnliinial leHliiiji 

Servit'e 

Tanop.hk^v, liolu'rl, Metnipnlilati iJfr 

iosurnnre C'pmpnn^ ' 
Tanhev, ticrlrude, bakewcHid (N. JJ 

Senior High School 
Taylor, Jnslints Educational TrHtirig 

Service ' ,-\ 

TAYLon, Saiiiuel J., I'liiladclphia Dejjarl-^ 

nif'til of !*erHonii*'l 
Terkeim Mt^yer, _ VcHliiva I Jniv<'rHil.y, 

New VorHl City 
TEnMAi,, J. !^., I'jdiiJ'a! if»n(il n'Hlitig 

Service ij 
riiOMAH. NVilhaiM 1-., CnivfrHity nf \\ iH= 

conHin 

TmiRNrMKE, Ibfberl L., Teachers ( '.ullhp', 
f^oUmibin I'nivc^rsity ' ^ 

riNKij', Jack W., Mitchel Air Varrr 
Base » . 



Stein, Morris L, University of C'hicago Tonnr:, MiJltrain, (^(jlunibia IJiuNersity 
Stephan, Frederick F., Prin*^et<ifs Uni- I raii.. Stariley Nt . lUii>de Inland ("cillego 

vernity * Of Edncation 

Stewart, Mary, lustittite of PhyHical Thaxler, Arthur 11., I^dueaticmal 

Medicine and ne^habilitation, N*^w Ib'cordH Bureau 



York University 
SriCE, Glen, Edueatiojial Teating {Service 
Stodoia, Quetitin C., b^ducatiiHial TeHl- 

itig Service 
SroKEH, Thomas M., Metropolitan Life 

Insurance Company 
STo^fE, Paul T., Huntingdon College, 

Montgomery, Ala. 
SrfHjGHTON, Robert W., Connecticlit 

State Departmeitl of Educatioti 



■rnKifJH, Krancea, (jsmmittee on Diag- 
noHtic Ht^ading TentR, Inc=, New York 
f:ity 

I'lKtKER, I^Hiyftrd ft, EiiucflticJiiftI Tei^t- 

irig Service < 
TLfRNBULL, VN illinni SS fuhiralional 

Testing Service ^ 
rYi.Erf, l^^ona LJ., University of Oregon, 
llnBArv, Ibigh B., Pc^nnsylvanla Stnte 

I 'niversity 



125. 



^^2 



i M7 ImM iortal Confe rence on Testing Problems 



fjRQUHAftf, Hdon hn Drown Universily 

Valentine, John A„ Educational Test- 
ing Service 

Valley, John R., Educational Testing 
Service > '■ 

ViGKERY, K. N., Clomson College, 
' Clomiotii S, C. 

Vctes, Harold A., Oflice of Naval Re- 
search, *Port Waihingtori, N. Y. 

Wadell* Blandenpi C, World^ Book 
Cpmpany 

Wa6ner, E. Paul, State Teachers CoU 

lege, Bloomsburgi Pa. 
WAHLORSNi Hardy L„ State Teachers 

CoUegei Geneseo, Y. 
Waldman, John, Pace College, New 

York^ity 
. Wallace, Wimbum L., The Psycho- 
- - logical Corpocation y \ 
WALiEBj Raymond L,, Allentown (Pa.) 

Public Schools 
Wallmark, Madeline, Educational Teat- 

ing Service 
Walsh, John J:, Boston College 
Walton, Wesley W., Educational Test- 
■ i \ ing Service « 
jyAHD, Anrtie W,, University of Ten- 
:\ nessee 

Wi^kTKiNSs Betty Jm Educational Testing 

Service ^ 
WATHiNe, Richard W,, Educational Test- 
. ing Service 
Watson, W^alter S^The Cooper Union 
Webster, Harold/ Vasaar College 
;Weiss, Eleanor S|, Educational Testing 
Service 

Weiss, Joseph, Por^chnic Institute of 

Brooklyn / 
Weitz, Henry, Duke UniverHity 
Weitzslvn, Ronald, Educational TchI- 

ing ^rvice 
Wellck, a. A,, Univecsity of New 

-Mexico 

Wesman, Alexander G„ The PByeho- 

iogical Corporation 
West, Elmer D., American Council on 

Education 
WmGHAM, E. L,, JtVilmington (Del.) 

Board of Pj^lic Educatioii 



WiilTNKY, Alfred G., Life Insurance ^ 
Agency Management A^sociatmn 

Wiener, Solomon, New York City De- 
partmunt of Personnel 

WiLKE, Marguerite M,, Greenwich;. 
(Conii^^ Board of Education 

WiLKE, Waller lU New York University 

WiLKiNs, Walter L,, St. I^Hiis University 

WiLKs, S, 3,, Princeton University 

\NiLLAnD, Richard W., Massachusetts 
Institute of Technology 

WiLLEMfN, Louis p.. Department of the 
Army \ . 

WiLLEY, Clarence F., Norwich Uni- 
versity, Northfield, Vt 

Williams, Roger K., v Morgan State 
College* Baltluiore, Md. 

Wii^soN, Kenneth M., Princeton Uni- 
versity 

Wilson, Robert G, Reed College, Port^ 
landi Ore. 

yViNANS, S, David, New Jei^ey State 

Department tif liducation 
WiNGO, Alfred L., Virginia State Dupart- 

nient of Education * 
\N'iNiEW!C:^, C. S., U, S. Naval Examin- 
ing Center, Great^ I^akes, IIU 
WiNTRnBOTTOM, John A., Educational 

Testing Service 
Winsin, Jane, Educational Testing 

Service - 
WinsiG, Woodrow, Educational Testing 

Service _ ^ 
WoMEi\, Frank University of 

ISIichigan 

Wt)OD, Hay G.% Ohio State Department 
of Education . 

WooDiiLiUY, Max A., Now York Uni- 
versity 

WoOLLATT, Ijorne H„ Baltimore (MdO 

Public Schools 
Wright, Wilbur H., State Teachers CoU 

lege, Geneseo, N, Y* 
WniGimTONis, J. Wayne, New York 
. City Board of Education 
Yatics, Vivian M., New York City 
YouNO, Donna S„ University of South 

Cafolina 

Sheldon S., New York Uni- 

yersity 

126 



^^3 



ERIC 



&tDN£R, Josepb, Depwtment of the ZuBm^ Jescph, New York State Pay- 
Array chiatrio Institute, New York City 

ZBtOLEn, Martin L., Pennsylvaniri Stnta v * 
, University 




D2tRl.S 



127 



ERIC 



